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1.0  Executive  Summary 


A  decision  science  working  group  was  chartered  in  Sep  2002  by  the  Information 
Directorate  (AFRL/IF)  to  assess  the  state-of-the-art  in  the  application  of  decision 
support  science  and  technology  (S&T);  and  to  recommend  ways  to  infuse  the  latest 
technologies  and  methodologies  into  the  Air  Force’s  Information  Systems  S&T  portfolio. 
The  initial  objective  statement  for  the  team  reads  as  follows: 

The  underlying  goals  of  the  DSWG  are  to  internalize  (within  AFRL)  best  practices  in 
the  application  of  Decision  Science  (DS),  and  to  drive  advancement  in  the  state-of- 
the-art  to  meet  AF  needs.  (AFRL/IF  working  group  presentation,  Sep  2002) 

One  of  the  first  actions  of  the  IF-Directorate  team  was  to  enlist  the  support  of  experts 
within  AFRL’s  Human  Effectiveness  Directorate  (AFRL/HE).  Over  the  ensuing  months 
and  through  a  series  of  coordination  meetings,  it  was  determined  that  a  joint  IF-HE 
working  group  should  address  Air  Force  decision-support  science  and  technology 
needs.  This  report  summarizes  the  activities  and  findings  of  that  joint-directorate 
Decision  Science  Working  Group  (DSWG). 

By  Feb  2003  the  joint  DSWG  team  had  established  and  agreed  upon  the  following 
terms  of  reference: 

Goal:  To  ensure  that  AFRL  has  and  applies  the  necessary  competence  in 
Decision  Science  to  address  Air  Force  needs. 

Toward  achieving  these  goals,  the  DSWG  will  attempt  to: 

•  Assess  the  state-of-the-art  in  the  development  of  Decision  Support 
Systems. 

•  Investigate  ways  to  internalize  best  practices  in  the  application  of  Decision 
Science  (DS)  within  our  respective  directorates. 

•  Explore  how  DS  can  help  address  Air  Force  needs. 

•  Focus  on  C4ISR. 

•  Focus  on  a  time-frame  beginning  with  the  near-term  and  extending  toward 
JV2020. 

•  Recommend  changes  to  our  S&T  “Roadmaps”  to  emphasize  opportunities 
in  DS  leading  toward  decision  superiority. 

•  Institute  a  process  that  promotes  collaboration  between  IF  and  HE  to 
satisfy  technology  needs  and  ultimately  deliver  more  effective  Decision 
Support  Systems  (DSS)  to  warfighters. 

During  2003,  the  DSWG  consulted  with  a  diverse  group  of  experts  from  government, 
industry  and  academia  through  workshops,  site-visits,  personal  interviews,  conferences, 
and  guest-lecturer  events.  The  DSWG  identified  a  broad  array  of  relevant  fields  of 
study  ranging  from  human-centric  social,  psychological,  behavioral  and  cultural 
sciences  to  more  computationally-focused,  computer  science  and  decision  theoretic 
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sciences.  These  disciplines  each  have  a  unique  research  community,  domain  ontology, 
terminology,  theory,  set  of  underlying  assumptions,  and  perspective.  The  breadth  of 
perspectives  and  their  respective  potential  contributions  underscore  the  need  for 
interdisciplinary  approaches. 

Although  the  working  group  fell  short  of  achieving  all  the  stated  goals,  the  DSWG  is 
pleased  to  report  a  number  of  significant  accomplishments: 

(1 )  A  significant  numbers  of  the  AFRL  S&T  staff  were  exposed  to  leading  theories 
and  research  in  decision  making  and  decision-support  technologies,  and  were 
given  an  opportunity  to  consult  with  world  leaders  in  decision  science. 

(2)  The  majority  of  these  DSWG  activities  were  conducted  cooperatively  between 
the  Information  (IF)  and  Human  Effectiveness  (HE)  directorates  of  AFRL.  As  a 
result,  the  degree  of  interaction  between  the  two  directorates  has  been 
significantly  elevated,  yielding  greater  understanding  and  the  identification  of 
numerous  opportunities  for  future  collaboration  in  the  development  of  decision 
support  systems,  DS  research,  and  exploring  decision  support  technologies. 

(3)  The  activities  of  the  DSWG  contributed  directly  to  the  development  of  a 
noteworthy  RAND  Report  entitled  “Implications  of  Modern  Decision  Science  for 
Military  Decision-Support  Systems,”  Project  Air  Force,  2005. 

(4)  A  number  of  specific  research  projects  have  emerged  from  the  work  group 
activities,  each  in  its  own  right  extending  our  understanding  of  decision  science 
and/or  making  advances  in  key  underpinning  technologies  of  decision  support 
systems. 

In  light  of  the  DSWG  not  fully  achieving  all  of  the  stated  objectives,  and  an  internal 
assessment  by  the  working  group  that  the  activities  undertaken  have  proven  beneficial 
to  AFRL,  a  few  follow-up  actions  are  recommended: 

(1 )  A  multi-disciplinary  decision  science  research  and  technology  group  like  the 
DSWG  should  be  continued  to  maintain  the  momentum  that  has  been 
established  in  strengthening  the  ties  across  the  AFRL  decision  science 
community. 

(2)  The  activities  sponsored  by  the  DSWG  to  increase  AFRL  staffs  awareness  and 
understanding  of  decision  science  research  were  very  well  received.  With  or 
without  a  DSWG,  these  activities  should  be  continued  and  encouraged: 
sponsoring  focused  workshops  in  DS-related  topics,  hosting  expert  lecturers  at 
AFRL,  and  conducting  site  visits  to  maintain  an  awareness  of  what  other  experts 
are  doing  in  this  field. 

(3)  A  list  of  DS-related  research  topics  was  developed  and  is  recommended  as  a 
starting  point  for  shaping  AFRL’s  research  and  development  portfolio  in  decision 
science  (see  section  5.0). 

Drawing  on  the  collective  core  competencies  of  both  AFRL/HE  and  AFRL/IF  together 
should  foster  a  richer  understanding  of  decision  science,  and  in  time,  provide  the  Air 
Force  a  more  robust,  capable,  and  intuitive  family  of  decision-support  systems. 
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2.0  Introduction 

The  Decision  Science  Working  Group  (DSWG)  functioned  as  an  informally  structured 
team  of  AFRL  researchers  active  during  the  latter  part  of  2002  and  throughout  2003. 

The  team  was  formally  chartered  to  investigate  the  broad  field  of  decision  science  (or 
sciences),  and  to  take  actions  to  ensure  AFRL  draws  upon  and  applies  the  best 
technologies  and  practices  in  the  application  of  decision  science  (DS)  in  the 
development  of  decision  support  systems  (DSS)  to  meet  current  and  future  AF  needs. 

This  report  summarizes  the  work  of  the  DSWG  starting  with  introductory  and 
background  information  in  section  2.  In  section  3  the  principal  activities  undertaken  by 
the  team  are  described.  In  section  4  the  working  group’s  accomplishments  are 
summarized,  and  in  section  5  a  set  of  conclusions  and  recommendations  are  presented. 
Finally,  in  section  6  a  list  of  references  are  provided. 

2.1.  Motivation 

Decision  Making  is  a  key  enabler  for  Information  Superiority  as  defined  in  Joint  Vision 
2020.  Potential  users  of  decision  support  systems  include  analysts,  planners,  and 
warfighters  from  all  services  and  levels  of  responsibility.  Both  AFRL’s  Information  and 
Human  Effectiveness  directorates  are  making  significant  investments  in  understanding 
and  extending  decision  support  science  and  technology,  and  developing  new  decision- 
support  capabilities  to  improve  Air  Force  decision  making. 

One  of  the  key  motivations  for  establishing  a  DS  team  within  the  Information  Directorate 
grew  from  observations  made  by  a  2001  Air  Force  Scientific  Advisory  Board  (SAB) 
review  of  the  directorate’s  Dynamic  Planning  and  Execution  thrust.  The  most  relevant 
concern  and  recommendation  voiced  from  that  review  follows: 

’’The  C2  Decision  area  was,  surprisingly,  missing  any  obvious  decision  science  and 
theory  component. 

•  Area  seems  overly  skewed  towards  modeling  and  simulation 

•  Broaden  investigation  areas  to  cover  decision  quality,  decision  timeliness,  etc.” 

In  response  to  these  observations,  as  well  as  other  concerns  held  by  AFRL  leadership, 
it  was  determined  that  a  dedicated  team  should  be  established  to  investigate  more 
thoroughly  the  current  state-of-the-art  in  decision  science,  and  within  that  context,  the 
suitability  of  the  AFRL  science  and  technology  portfolio.  The  team  that  was  formed, 
their  charter,  and  the  activities  undertaken  are  described  in  the  sections  that  follow. 


2.2.  DSWG  Charter 

The  Information  Directorate  team  was  established  in  Sep  2002  with  the  following 
objectives: 
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•  Determine  the  requirement  for  the  application  of  decision  support  science  & 
technology  in  the  context  of  AFRL/IF’s  broad  S&T  portfolio. 

•  Assess  the  state-of-the-art  in  the  application  of  decision  support  science  & 
technology. 

•  Infuse  decision  support  technologies,  methodologies  and  tools  in  the  exploration, 
development,  and  evaluation  of  AFRL/IF’s  decision  support  products. 

•  Recommend  changes  to  our  S&T  portfolio  and  culture  to  emphasize 
opportunities  in  decision  support  science  &  technology. 

One  of  the  first  activities  of  the  Information  Directorate  team  was  to  consult  with  experts 
in  the  field  of  human  decision-making  from  the  AFRL  Human  Effectiveness  Directorate 
(AFRL/HE).  Based  on  common  concerns,  the  degree  of  shared  interests  among  the 
two  directorates,  and  recognizing  an  opportunity  to  leverage  personnel  with 
complementary  technical  expertise,  it  was  determined  that  a  joint  directorate  (IF  and 
HE)  multi-disciplinary  DSWG  should  be  established  to  study  this  field. 

Working  as  a  cross-directorate  team,  by  Feb  2003  the  DSWG  had  established  the 
following  working  charter: 

Goal:  To  ensure  that  AFRL  has  and  applies  the  necessary  competence  in 
Decision  Science  to  address  Air  Force  needs. 

Toward  achieving  these  goals,  the  DSWG  will  attempt  to: 

•  Assess  the  state-of-the-art  in  the  development  of  Decision  Support 
Systems  (DSS). 

•  Investigate  ways  to  internalize  best  practices  in  the  application  of  Decision 
Science  (DS)  within  our  respective  directorates. 

•  Explore  how  DS  can  help  address  Air  Force  needs. 

•  Focus  on  C4ISR. 

•  Address  Air  Force  needs  from  the  near-term  through  JV2020  time-frame. 

•  Recommend  changes  to  our  S&T  “Roadmaps”  to  emphasize  opportunities 
in  Decision  Science  (DS)  leading  toward  decision  superiority. 

•  To  institute  a  process  that  promotes  collaboration  between  IF  and  HE  to 
satisfy  technology  needs,  ultimately  to  deliver  more  effective  DSS  to 
warfighters. 


2.3.  Team  Organization  /  Membership 

The  DSWG  was  established  as  a  partnership  between  AFRL/HE  and  AFRL/IF,  co-lead 
by  LtCol  Cindy  Dominguez  (HEX),  and  Mr.  Jerry  Dussault  (IFSE).  A  complete  list  of  the 
DSWG  membership  is  provided  below  (see  Figure  1 .) 
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Information  Directorate  (IF) 


Human  Effectiveness  Directorate  (HE) 


Dr.  Thomas  Triscari*  (IF) 
Joseph  Carozzoni  (IFSB) 
Maj  George  Tadda  (IFS) 
Mr.  Paul  Yaworsky  (IFSB) 
Mr.  Jerry  Dussault  (IFSE) 
Dr.  Kirk  Weigand  (IFSD) 
Dr.  Don  Gossink**  (IFT) 

*  IPA  Rensselaer  Polytechnic  Inst. 

**  Visiting  Researcher,  DSTO,  Australia 


Dr.  Janet  Miller  (HECA) 
LtCol  Cindy  Dominguez  (HEX) 
Mr.  Donald  Monk  (HECA) 
Dr.  John  Reising  (HEC) 

Mr.  Samuel  Kuper  (HESS) 
Dr.  Kevin  Gluck  (HEA) 

Lt.  Mona  Stilson  (HESR) 
Capt.  Brian  Tidball  (HESR) 
LtCol  Mark  Sheehan  (HECA) 


Figure  1.  AFRL  DSWG  Membership 


2.4.  Summary  of  Planned  Activities 

The  DSWG  developed  the  following  outline  of  planned  activities  early  in  2003: 

•  Invite  noted  Decision  Science  Experts  to  lecture  at  AFRL-sponsored  events 

•  Sponsor  workshops,  symposia,  seminars,  and  short-courses  focusing  on  Decision 
Science  topics  of  interest 

•  Make  Decision  Science  a  special  emphasis  subject  for  AFRL's  Information  Institute, 
for  FY03 

•  Team  with  AFRL/HE,  AFOSR  and  others  as  identified,  to  leverage  resources  and 
world-class  expertise 

•  Develop  a  knowledge  base  of  current  Decision  Science  activities  within  AFRL,  other 
service  labs,  across  Government  agencies,  industry,  and  academia. 

•  Make  site  visits  to  establishments  highly  regarded  for  their  Decision  Science 
expertise 

•  Supplement  AFRL  staff  with  Decision  Science  experts  from  academia,  (full-,  part- 
time,  sabbatical,  post-doc,  etc.) 

The  specific  actions  undertaken,  the  findings,  and  recommendations  from  the  working 

group  are  described  in  the  sections  that  follow. 
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3.0  Summary  of  Activities  Undertaken 

Beginning  in  2002  and  continuing  into  2003  the  DSWG  carried  out  an  array  of  activities 
to  address  the  goals  identified  above.  The  activities  included: 

•  sponsoring  and  participating  in  workshops,  conferences  and  seminars 

•  sponsoring  guest  lecturers 

•  visits  to  some  of  the  leading  researchers  in  decision  science 

•  sponsoring/directing  decision-science  research  projects 

These  activities  are  described  in  greater  detail  in  the  remainder  of  this  section. 

3.1.  Workshops ,  Conferences,  &  Seminars 

AFRL’s  DSWG  members  sponsored,  or  participated  in  a  number  of  workshops, 
conferences,  covering  the  field  of  decision  science.  The  most  noteworthy  events  are 
summarized  below. 

3.1.1.  DSWG  Kickoff  Meeting,  23-24  Oct  2002  at  GMU 

The  first  significant  event  sponsored  by  the  DSWG  was  a  kickoff  meeting,  held  23-24 
Oct  2002  at  the  Center  of  Excellence  in  Command,  Control,  Communications,  and 
Intelligence  (C3I)  at  George  Mason  University  (GMU),  Fairfax,  VA.  The  purpose  of  this 
meeting  was  to  bring  together  and  focus  research  and  development  (R&D) 
representatives  from  the  military,  academia,  and  industry  to  discuss  requirements, 
share  perspectives  and  pool  knowledge.  Specific  goals  were  to  define  “Decision 
Sciences,”  as  seen  from  varying  perspectives  and  particularly  in  the  context  of 
enhancing  military  decision  making  (Ref.  JV2020  and  AF  Vision  2020),  and  to  develop  a 
"Way  Ahead"  action  plan  for  AFRL  Information  (IF)  and  Human  Effectiveness  (HE) 
Directorates. 

Speakers  for  the  kickoff  meeting  were  selected  based  upon  their  experience  and 
reputation  as  leaders  in  fields  supporting  decision  science,  and  for  the  unique  expertise 
they  could  contribute  to  improving  our  collective  understanding  of  this  complex  field. 

The  list  of  invited  speakers,  their  affiliation,  and  the  title/subject  of  their  presentation 
follows: 

•  Dr.  Eduardo  Salas  (Univ.  of  Central  Florida),  “The  Science  of  Team  Performance 
in  Context:  Progress  &  Challenges” 

•  Dr.  Jean  MacMillan  (Aptima),  “Theory  and  Models:  Command  and  Control 
Teams” 

•  Dr.  Laurel  Allender  (ARL),  “Computing  Cognition:  The  Requirement  for  Models 
of  Commanders  Making  Decisions” 

•  Dr.  Dennis  Leedom  (Evidence  Based  Research),  “Organizational  Sensemaking” 

•  Dr.  Richard  Deckro  (AFIT/ENS),  “Decision  Sciences  in  Information  Operations” 

•  Dr.  Gary  Klein  (Klein  Assoc.),  “How  Information  Technology  Makes  People 
Stupid” 
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•  Dr.  Andrew  Sage  (GMU,  Dept  of  Sys  Engr  and  Ops  Research),  “Complex 
Adaptive  Issues  and  Considerations  in  Support  of  Decision  Making” 

•  Dr.  Marek  Druzdzel  (Univ  of  Pittsburgh),  “Augmenting  Human  Decision  Making 
Through  Normative  Systems” 

•  Dr.  Eugene  Santos  (UConn),  “Overview  of  Efforts  at  the  Intelligent  Distributed 
Information  Systems  Laboratory” 

•  Dr.  Lee  Ehrhart  (MITRE  Corp.),  “Cognitive  Systems  Engineering:  Decision- 
Centered  Design  and  Evaluation” 

•  Dr.  Jonathan  Kulick  (RAND),  “Decision  Sciences  at  RAND” 

A  complete  list  of  participants  is  provided  in  Appendix  A. 

A  few  key  take-away  concepts  from  the  kickoff  meeting  were: 

•  Need  for  a  multidisciplinary  approach  to  address  decision  science:  AFRL/HE 
and  AFRL/IF  complement  each  other. 

•  Issue  of  metrics  in  decision  making,  measures  should  be  output  based  rather 
than  input  or  activity-based.  Output  metrics  (e.g.,  decision  quality  or  timeliness) 
are  more  difficult  to  define  and  not  what  we  normally  measure.  Process  metrics 
are  important  too:  where  we’ve  had  an  impact  on  the  process,  and  identifying 
where  the  leverage  points  are. 

•  Need  to  balance  support  between  novice  and  expert  decision-makers. 

Encourage  novices  to  become  experts. 

•  Avoid  the  pitfalls,  things  we  do  with  information  technology,  that  make  users 
“stupid.”  Recognize  how  technology  influences  process  and  decision  making 
(both  positive  and  negative).  Detect  the  ways  that  proposed  information 
technology  projects  can  interfere  with  expertise. 

•  Use  Cognitive  Systems  Engineering  (CSE)  methods  to  create  a  good  fit  between 
information  technology  and  decision  makers. 

•  A  Decision-Centered  Design  approach  to  cognitive  engineering  can: 

o  ID  the  cognitive  requirements  of  the  work 
o  Describe  the  decision  strategies  to  be  supported 
o  Determine  the  metrics  for  gauging  success. 

•  Cognitive  Systems  Engineering  is  needed  at  three  points: 

o  At  the  beginning,  in  the  form  of  design  criteria 
o  During  the  design  process,  to  shape  the  system 
o  At  the  end,  as  Test  and  Evaluation  criteria. 

•  The  need  to  recognize  and  differentiate  between  different  classes  of  problems 
and  coping  strategies:  simple,  complex,  and  wicked. 

•  The  power  of  “Story  Frameworks”  to  communicate  ideas  to  decision  makers. 

•  Consider  four  types  of  cognitive  decision-making  models  with  different 
strengths/weaknesses,  all  still  “works  in  progress” 

o  Goal  Oriented 

o  Recognition-primed  decisionmaking  (RPD) 
o  Information  Theory 
o  Cognitive  Dynamic 
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•  DSWG  should  engage  the  US  Army’s  Advanced  Decision  Architecture  (ADA) 
Collaborative  Technology  Alliance  (CTA),  sponsored  by  Army  Research 
Laboratory. 

•  Theories  of  shared  cognition/situation  awareness  are  not  deep  enough 
(templates  not  detailed  enough). 

•  Theory  of  shared  mental  models.  Drivers:  Shared  mental  models,  shared 
cognition,  models  of  team  performance  (inputs,  outputs  and  throughputs), 
naturalistic  decision  making  (NDM). 

•  Teams  in  the  Wild  (naturalistic),  researchers  need  to  study  decision-making 
teams  in  their  natural  setting  -  how  they  really  make  decisions. 

3.1.2.  Decision  Science  Workshop,  15-18  Apr  2003  at  the 
Minnowbrook  Conference  Center 

In  April  2003  the  DSWG  with  AFRL’s  Information  Institute  (II),  co-sponsored  a  workshop 
on  decision  science;  the  workshop  was  held  15-18  Apr  2003  at  the  Minnowbrook 
Conference  Center  in  Blue  Mountain  Lake,  NY.  The  workshop  focused  on  identifying 
potential  research  and  development  investment  opportunities  in  the  science  and 
technologies  that  underpin  predicting  the  behaviors  of  individuals  and  military 
organizations,  in  support  of  Predictive  Battlespace  Awareness  (PBA)  and  the  decision¬ 
making  needs  of  the  Joint  Force  Air  Component  Commander  (JFACC).  Emphasis  was 
placed  on  exploring  how  Modeling  and  Simulation  (M&S)  technologies  could  be  used  to 
assist  PBA,  looking  across  the  spectrum  from  requirements  to  calibration  and  validation. 
The  specific  objectives  for  the  workshop  were  to: 

•  Refine  AFRL’s  vision  for  a  JFACC  training,  rehearsal  and  course  of  action  (COA) 
evaluation  environment. 

•  Recommend  an  R&D  "way  ahead"  for  exploratory  development. 

•  Identify  and  clarify  those  challenge  problems  that  warrant  additional  basic 
research. 

•  Promote  a  common  understanding  of  and  appreciation  for  concepts  such  as: 

PBA,  Effects-Based  Operations  (EBO),  Intelligence  Preparation  of  the 
Battlespace  (IPB),  and  Courses  of  Action  (COAs). 

•  Explore  interoperable  friendly  COA  (fCOA)  and  enemy  COA  (eCOA). 

The  workshop  participants  were  organized  into  three  focus  groups  to  explore  M&S 
technologies  that  support  and/or  advance  PBA: 

1.  Modeling  and  prediction  of  human,  aggregate,  organizational,  and  cultural 
behavior,  including: 

•  Human  behavioral  representation  (HBR)  technologies 

•  Prior  beliefs/perceptions/judgment/ideological/etc. 

•  Predictive  psychological  and  social  models 

•  Verification  and  validation  of  models 
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2.  Integration  of  the  human-element  model  into  the  information/geophysical 
models,  including: 

•  Integrating  the  various  models: 

o  Cognitive  models  (e.g.  behavioral,  cultural,  perceptual,  etc.) 
o  Information  models  (e.g.  internet,  media  broadcasts,  newspapers,  etc.) 
o  Geophysical  models  (e.g.  physical  locations  and  objects  such  as  tanks, 
infrastructure,  etc.) 

•  How  to  generate  the  eCOAs 

3.  COA  vs.  eCOA  Simulation  and  Analysis,  including: 

•  Assessing  fCOA  versus  eCOA  as  an  integrated  process 

•  Assessing  multiple  friendly  COAs  against  multiple  enemy  COAs  (e.g. 
wargaming) 

•  How  to  simulate  forces  that  react  intelligently  to  friendly  actions 

•  Calibration 

•  Decision  and  visualization  support 

•  Validation  and  verification 

A  complete  list  of  workshop  participants  is  provided  in  Appendix  B. 

The  results  from  the  three  focus  groups  listed  above  were  presented  in  a  general 
session.  At  that  point  it  was  determined  that  it  would  be  prudent  to  mix  and  cross- 
fertilize  the  membership  of  the  workgroups,  and  to  proceed  by  refocusing  the 
workgroups  on  a  few  specific  areas  of  concern  to  CPE,  namely:  (1 )  Effects-Based 
Operations  (EBO),  (2)  Course  of  Action  (COA)  development,  and  (3)  COA  analysis  and 
selection.  The  three  groups  were  given  identical  tasking.  They  were  asked  to  work  in 
parallel  breakout  sessions  to  respond  to  the  following  challenges: 

•  Identify  opportunities  to  make  Air  Operations  Center  (AOC)  warfighters  more 
effective, 

o  Identify,  describe  and  prioritize  up  to  5  suggestions, 
o  Emphasize  capabilities  that  would  have  a  positive  impact, 
o  Be  forward  thinking,  the  time  frame  is  the  next  3-10  years. 

•  Identify  approaches  to  achieve  the  capability  and  to  overcome  the  technical 
challenges  (i.e.,  why  is  it  difficult). 

The  results  from  these  final  three  breakout  groups  were  briefed  to  all  the  participants, 
and  the  recommended  topics  are  summarized  below. 

Group  #1  ( Main  Lodge)  recommendations: 

•  Capability  1:  Constraint-aware,  continuous  COAs. 

•  Capability  2:  Team  Decision  Making  Aids  (TDMA);  emphasizing  support  to 
distributed  teams. 

•  Capability  3:  Intelligent  Critic  for  Problems  Solving  and  Decision-making 
Personalization. 
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Group  #2  ( Boathouse )  recommendations: 

•  Capability  1 :  Interoperable  Framework  of  COAs;  a  capability  that  allows  fCOA 
and  eCOA  games  to  be  played  out.  Scenario  generation  with  real  world  data.  A 
new  generation  of  technology  beyond  simply  attrition-based  war-gamming. 

•  Capability  2:  Explanation/rationale,  reporting/analysis  process  for  COA  analysis. 
“Point  and  click”  drill  down.  COA  comparison  and  a  capability  to  visualize  data 
beyond  the  typical  temporal  and/or  spatial  presentations. 

•  Capability  3:  Constraint  aware  COA  development  (e.g.,  logistics,  legal,  financial, 
etc.). 

•  Capability  4:  PSYOPS  (non-kinetic)  integration  into  the  COA  process  (e.g., 
social,  political,  cultural,  and  “the  press”). 

•  Capability  5:  Continuous  COA,  “the  ultimate  goal.”  Continuous  analysis, 
refinement,  and  assessment  (operational  assessment),  fed  into  the  COA 
development  process. 

Group  #3  ( Classroom )  recommendations: 

•  Capability  1 :  Over  5  years,  systematically  build  low-resolution  phenomenology- 
based  model  structures  to  inform  automated  metamodeling  and  calibration  from 
detailed  models  and  other  “data” 

•  Capability  2:  Refine  concept  of  exploratory  analysis  and  develop  first-generation 
sophisticated  tools  for  accomplishing  it  in  JTF  and  JFACC  contexts. 

o  Find  robust,  adaptive,  COAs  very  likely  to  move  us  in  “right  direction” 
without  violating  constraints. 

o  As  part  of  this,  automatically  track  points  of  Red  and  Blue  vulnerability 
(critical  components)  to  suggest  ways  to  hedge  and  improve  robustness. 

•  Capability  3:  Review  and  redefine  “validation”  for  low-res  models  used  in 
exploration.  Validity  relates  to  structure  and  parameter  ranges,  not  predictive 
accuracy  per  se. 

•  Capability  4:  Develop  tools  and  methods  to  assist  intelligent  aggregation. 

Correct  aggregation  involves  complex  “averaging”  over  cases,  parameter  values, 
etc. 

The  detailed  outbriefings  from  each  of  the  workgroups  were  organized  and  made 
available  to  all  the  participants  in  electronic  form. 


3.1.3. 12th  Conference  on  Behavior  Representation  in  Modeling  and 
Simulation  (BRIMS),  12-15  May  2003,  Scottsdale,  AZ. 

This  annual  event  provides  a  forum  for  scientific  and  technical  exchange  on  research  in 
the  science  of  human  behavior  representation  and  on  the  application  of  leading-edge 
cognitive  science  to  the  behavior  representation  challenges  faced  by  the  modeling  and 
simulation  community.  Application  areas  of  interest  include  training,  rehearsal,  analysis, 
acquisition,  planning,  experimentation,  and  gaming.  The  conference  enables  modeling 
and  simulation  research  scientists,  engineers,  application  users,  and  technical 
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communities  to  meet,  share  ideas  and  experiences,  identify  gaps  in  current  capabilities, 
discuss  new  research  directions,  and  highlight  promising  technologies. 

DSWG  and  AFRL  members  who  attended: 

•  Kevin  Gluck,  AFRL/HEA  (Conference  Chair) 

•  Joe  Carozzoni,  AFRL/IFSF 

•  John  Graniero,  AFRL/IF  (Information  Institute) 

•  Maj  George  Tadda,  AFRL/IFS 

A  key  conference  “take-away”  was: 

•  The  state-of-the-art  in  this  area  is  still  immature.  Much  of  the  work  presented 
was  in  the  computer  science  field,  as  opposed  to  human  understanding  and 
modeling  human  performance  fields. 

Recommendations:  The  areas  covered  were  much  broader  than  decision  sciences. 
While  some  benefit  was  received,  this  conference  is  not  likely  to  be  of  premier  interest 
to  those  focused  on  decision  sciences,  unless  there  is  a  pressing  secondary  interest  in 
modeling  and  simulating  human  behavior. 


3.1.4.  Tutorial  on  Cognitive  Task  Analysis  (CTA),  18  Mar  03,  at  AFRL, 
Rome,  NY. 

Cognitive  Task  Analysis  (CTA)  refers  to  a  broad  umbrella  of  techniques  and  methods 
that  can  be  employed  to  improve  the  usefulness  of  information  technology  systems  by 
emphasizing  key  cognitive  tasks.  CTA  entails  a  principled  process  for  understanding 
user  cognitive  requirements  and  work  constraints,  to  apply  towards  the  design  of 
systems.  In  this  tutorial,  the  speakers  introduced  and  explained  general  definitions  and 
discussed  the  impact  of  CTA  on  system  design.  The  speakers  described  several 
families  of  CTA  methods  and  presented  examples  illustrating  how  they  have  been  used. 

Various  methods  for  capturing  traceable  customer  requirements  were  reviewed, 
including  concept  maps,  abstraction  hierarchy,  Applied  Cognitive  Task  Analysis  tool 
(including  knowledge  audit  methodology),  knowledge  elicitation  interview  techniques, 
simulation  interviews,  participatory  design  and  other  tricks  of  the  trade.  An  in-depth 
exemplar  was  discussed,  to  illustrate  how  CTA  has  been  used  to  develop  a  Work- 
Centered  Support  System  for  Air  Mobility  Command.  This  Global  Weather 
Management  system  has  helped  AMC  flight  managers  and  weather  forecasters  avoid 
costly  delays  and  minimize  other  potential  impacts  to  tanker  and  airlift  operations. 
Thorough  mining  and  analysis  of  cognitive  requirements  helped  the  computer  and 
cognitive  scientist  team  develop  an  information  technology  system  of  great  usefulness 
to  AMC. 

The  tutorial  was  arranged  for  AFRL/IF  by  AFRL/HE,  and  the  presentations  were  given 
by: 

•  Lt  Col  Cindy  Dominguez,  AFRL/FIEX 
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•  Dr  Janet  Miller,  AFRL/HEC 

•  Mr.  Samuel  Kuper,  AFRL/HES 

•  Ms.  Laura  Militello,  (contractor)  AFRL/HES R 

An  expanded  list  of  AFRL/IF  participants  is  provided  in  Appendix  C.  A  complete  set  of 
the  presentation  charts  and  references  were  made  available  to  all  the  tutorial  attendees. 
An  extensive  CTA  bibliography  was  provided,  and  is  included  in  Appendix  D. 


3.1.5.  Cognitive  Systems  Engineering  (CSE)  Seminar;  5-7  Aug  2003,  at 
the  White  Eagle  Conference  Center,  Hamilton,  NY. 

The  purpose  of  this  three  day  seminar  was  to  train  AFRL  scientists  and  engineers  on 
processes,  tools,  and  techniques  of  Cognitive  Systems  Engineering  (CSE),  and  to  foster 
the  adoption  of  best  practices  in  the  application  of  CSE  to  the  design  and  development 
of  C2  Decision  Support  Systems.  The  list  of  speakers  included  experts  from  academia 
and  industry  as  follows: 

•  Dr.  Robert  Hoffman,  Univ.  of  West  Florida, 

•  Dr.  Nancy  Cooke,  Arizona  State  University 

•  Mr.  Isaac  Brewer,  GeoVista  Center,  Penn  State  Univ. 

•  Dr.  Gary  Klein,  Klein  Associates 

•  Dr.  Wayne  Zachary,  CHI  Systems,  Inc. 

•  Dr.  Mica  Endsley,  SA  Technologies 

Each  of  the  six  speakers,  except  for  the  last  one,  was  given  a  three-hour  time  block  in 
which  to  present  and  conduct  interactive  activities  relative  to  their  chosen  topic,  in  a 
“learning-focused”  atmosphere.  Speakers  were  directed  to  plan  their  agenda  to  include 
interactive  activities  relative  to  their  presented  topic,  so  that  participants  could  get 
“hands  on”  experience  with  the  concepts  presented.  The  project  manager  coordinated 
the  planning  of  topics  and  activities  with  the  speakers  to  ensure  there  would  be  no 
duplication  of  topics  or  activities.  As  a  result  of  this  focus  and  coordination  the  following 
schedule  and  topics  were  identified  and  adopted  for  the  seminar. 


Date/Time 

Speaker 

Topic 

Tues.  AM 

Robert  Hoffman 

An  overview  of  CSE,  Concept  Mapping 

Tues.  PM 

Isaac  Brewer 

Establishing  a  Knowledge  Elicitation  Process 

Wed.  AM 

Gary  Klein 

Decision-centered  Design  Process 

Wed.  PM 

Mica  Endsley 

Designing  for  Situation  Awareness  in  Complex  Systems 

Thurs.  AM 

Nancy  Cooke 

Team  Cognition 

Thurs.  PM 

Wayne  Zachary 

Cognitive  Agent  Development 

Table  1.  CSE  Seminar  Agenda 


Participation  in  the  CSE  Seminar  was  by  invitation  only  and  was  planned  to  include 
representatives  from  across  the  divisions  and  branches  of  AFRL  at  the  Rome  Research 
Site,  as  well  as  some  members  from  the  Human  Effectiveness  (HE)  directorate  from 
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Wright-Patterson  AFB.  Attendance  was  limited  to  42  per  the  arrangement  with  the 
White  Eagle  Facility.  The  complete  list  of  participants  is  provided  in  Appendix  E. 

In  lieu  of  providing  a  hard  copy  notebook  of  briefings,  a  CD  (titled  CSE  Seminar 
Resource  Kit),  was  given  to  each  participant  and  speakers  at  the  beginning  of  the 
seminar.  In  addition  to  the  presentations,  it  contained  some  of  the  tools  referenced  by 
the  experts,  an  electronic  copy  of  the  HSIAC  SOAR  titled  “Cognitive  Systems 
Engineering  in  Military  Aviation  Environments:  Avoiding  Cogminutia  Fragmentosa!  “, 
and  several  articles  and  references  recommended  by  the  speakers.  Many  participants 
used  the  CD  during  the  seminar  to  follow  along  with  each  speaker.  At  the  request  of 
other  participants  some  hard  copies  of  the  presentations  were  provided  at  the  beginning 
of  each  session. 

The  seminar  was  organized  and  the  activities  coordinated  in  large  part  by  Ms.  Ellen 
Walker  of  ITT,  through  an  agreement  with  the  Data  Analysis  Center  for  Software 
(DACS).  A  summary  of  the  materials  presented  during  the  CSE  seminar  is  provided  in 
Appendix  F. 

3.1.6.  AFOSR  Decision  Sciences  Workshop,  16- 17  June  2003,  George 
Mason  University 

The  purpose  of  the  workshop  was  to  evaluate  current  research  activities  and  identify 
areas  in  Decision  Sciences  that  warrant  further  investment  in  basic  research.  Dr.  Bob 
Sorkin/AFOSR  opened  the  workshop  stating  the  objectives  were  to: 

•  Explore  the  most  effective  use  of  science  in  both  the  human  and  computer 
domains  to  advance  the  development  of  decision  support  systems. 

•  Provide  opportunities  for  AFOSR,  HE,  and  IF  to  integrate  their  approaches  and 
define  ways  to  ensure  that  the  development  and  application  of  science  from 
these  areas  is  working  effectively  towards  a  common  goal. 

Day  one  of  the  workshop  consisted  of  two  parts.  During  the  first  part  of  the  day, 
presentations  were  given  by  members  of  the  AFRL/HE,  /IF  and  AFOSR/NL  and  /NM 
directorates,  outlining  the  current  status  of  work  on  decision  support  systems.  The 
second  part  consisted  of  five  working  groups  led  by  members  of  the  expert  panel.  On 
the  second  day  briefings  were  presented  by  each  of  the  working  groups,  followed  by  a 
final  session  to  integrate  the  issues  identified  by  each  of  the  five  working  groups. 
AFOSR  indicated  that  they  had  reached  their  goal  of  having  something  ‘in  their  hip 
pocket’  should  funding  opportunities  arise.  Lists  of  both  the  research  areas  identified 
and  the  invited  attendees  are  provided  in  Appendix  G. 
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3.1.7.  6th  International  Conference  on  Naturalistic  Decision  Making 
(NDM6),  15-17  May  2003,  in  Pensacola,  FL. 

The  International  Natural  Decision  Making  (NDM)  conferences  serve  as  an  outlet  and 
forum  for  the  community  of  researchers  involved  in  the  study  of  problem  solving, 
decision  making,  expertise,  and  situational  awareness,  aimed  at  solving  real  world 
problems.  The  NDM  field  focuses  on  studying  real  decision-makers  in  their  natural 
settings.  There  was  a  call  for  developing  a  more  generalizable  theory  of  macro¬ 
cognition  and  decision-making.  This  year’s  conference  was  organized  by  Dr.  Robert 
Hoffman  of  the  Institute  for  Human-Machine  Cognition  (IHMC),  and  both  Army 
Research  Laboratory  and  AFRL  helped  to  underwrite  the  cost  of  the  conference. 
Selected  leading  scholars  and  researchers  in  the  field  presented  excellent  talks  and 
topical  sessions  that  included  relevant  case  examples  of  cognitive  work  and  task 
analyses,  military  examples  of  the  breakdown  in  decision-making,  situated  cognition, 
smart  heuristics,  the  study  of  expertise,  expert  apprenticeships,  macrocognitive  models 
in  the  design  of  complex  systems,  and  applying  NDM  to  real  world  needs. 

While  there  was  interchange  and  networking  with  attendees,  the  primary  benefit  of 
attending  this  event  was  listening  to  the  world-class  leaders  speaking  to  an  audience  of 
their  peers: 

•  Kim  Vincente  (University  of  Toronto),  special  guest  speaker,  discussed  a  case 
study  entitled  ‘A  Tipping  Point  in  NDM:  A  Case  Study  of  Radical  Cultural  Change 
in  Health  Care.’ 

•  Gerd  Gigerenzer ,  director  at  the  Max  Planck  Institute  for  Human  Development  in 
Berlin,  gave  a  talk  entitled  ‘Less  is  More:  How  Smart  Heuristics  Work.’ 

•  Scott  A.  Snook,  a  retired  U.S  Army  officer,  and  an  expert  in  study  of  military 
leadership  and  organizational  systems,  gave  a  talk  on  the  need  for  situational 
awareness  research  with  the  example  of  the  friendly  fire  shoot-down  of  two  U.S. 
Black  Hawks  over  Northern  Iraq. 

•  Earl  Hunt,  a  world  leader  in  experimental/cognitive  psychology,  discussed  an 
experiential  method  of  education  based  on  cognitive  science  and  the  studies  of 
schools  that  have  adopted  this  approach. 

•  William  J.  Clancey,  Chief  Scientist,  Human-Centered  Computing,  at  NASA-Ames 
Research  Center,  gave  a  talk  on  Situated  Cognition. 

•  Jennifer  Wiley,  Tom  Omerod,  and  James  Shanteau  discussed  expertise  out  of 
context. 

•  Emily  Roth,  Laura  Militello,  Valerie  Shalin,  and  Lia  Dibello,  and  Frank  Yates 
discussed  expert  apprenticeships. 

•  Gary  Klien,  Karol  Ross,  Brian  Moon,  Dave  Woods,  Judith  Orasanu,  and  Mica 
Endsley  discussed  the  role  of  Macro-cognitive  models  in  the  design  of  complex 
cognitive  systems. 

•  Jan  Maarten  Schraagen  and  Lt  Gen  Rick  Brown  (retired)  discussed  applying 
NDM  to  real  world  needs. 

This  was  an  excellent  conference.  The  talks,  topical  sessions,  and  poster  sessions  gave 
a  wealth  of  information  on  the  research  going  on  in  this  field  as  well  as  the  opportunity 
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to  network  and  make  contacts  with  those  doing  similar  research.  AFRL  participants 
included:  Dr.  Janet  Miller,  Dr.  Kirk  Weigand,  and  1  Lt  Mona  Stilson. 

Recommendations:  The  focus  of  this  conference  has  changed  somewhat,  from  a  pure 
naturalistic  decision-making  forum  to  a  broader  exploration  of  the  human  aspects  of 
decision-making.  However,  the  conference  does  not  address  computational  decision¬ 
making  issues  or  the  challenges  we  face  in  bridging  the  cognitive  and  computational 
research  communities.  Undoubtedly  the  conference  will  continue  to  be  interesting  and 
well-managed,  and  should  remain  on  the  ‘radar  scope’  of  the  AFRL  decision-science 
community. 


3.1.8.  Personality  and  Cultural  Factors  in  Military  Gaming  Workshop, 
9-10  July  2003,  DMSO,  Alexandria,  VA. 

The  Defense  Modeling  and  Simulation  Office  held  a  ‘Personality  and  Cultural  Factors  in 
Military  Gaming  Workshop’  to  determine  efficient  experimental  methods  for  isolating 
personality  and  cultural  predictors  in  performance  in  military-oriented  synthetic 
environments.  It  is  important  to  know  whether  personality  and  cultural  variables  predict 
performance  or  styles  of  play  in  synthetic  environments  having  a  military  focus.  The 
widening  variety  of  potential  military  operations  (e.g.,  combating  terrorism,  coalition 
warfare,  fighting  in  urbanized  areas,  and  homeland  defense)  makes  information  about 
human  action  and  reaction  more  salient  in  simulations  that  support  training,  doctrine 
development,  tactical  planning,  and  related  military  needs.  Threats  now  come  from 
highly  organized  bands  of  criminals  like  narcotics  traffickers,  paramilitary  fanatics,  and 
terrorists,  not  just  conventional  forces  with  traditional  military  markings  and  modes  of 
operation.  These  adversaries  present  “asymmetric”  threats  and  they  loom  large  in 
public  anxiety,  especially  since  September  1 1 , 2001 .  A  specific  goal  of  this  workshop 
was  to  develop  a  roadmap  for  new  empirical  research  to  include  cross-cutting  variables, 
potential  partnerships,  and  applications  in  M&S  toolsets.  A  framework  question  was: 

Do  measured  personality  and/or  cultural  variables  predict  performance? 

The  attendees  represented  government,  academic,  and  commercial  sectors.  On  the 
first  day,  candidate  theories  and  instruments  for  personality  and  cultural  assessment 
were  reviewed.  Important  applications  such  as  command  and  control,  and  adversarial 
decision  making  were  illustrated;  and  computer-based  games  (or  synthetic 
environments)  with  a  military  focus  were  discussed.  On  the  second  day,  breakout 
groups  formulated  strategies  for  gathering  empirical  data  linking  personality  and  cultural 
factors  with  wargame  performance  and  style  of  play.  Four  breakout  groups  were 
organized  around  the  following  topics: 

•  The  use  of  cultural  and  personality  data  in  representative  classes  of  military 
models  and  simulations. 

•  The  research  strategies  and  methods  used  for  producing  these  data. 

•  Candidate  synthetic  environments  in  the  commercial  marketplace  and 
government  locations  that  might  support  a  gaming  testbed  strategy. 
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•  Test  and  evaluation  criteria  for  determining  the  validity  and  utility  of  given 
research  designs  and  applications  of  data  in  military  modeling  and  simulation 
techniques. 

The  goal  was  to  achieve  a  consensus  on  which  personality/cultural  dimensions  are 
most  relevant  for  study,  which  instruments  are  most  useful,  and  how  these  might  be 
related  to  performance  sampled  through  wargame  simulators. 

Recommendations:  While  the  focus  of  discussion  included  considering  culture  and 
other  human  attributes  that  affect  decision-making,  much  of  the  discussion  centered 
around  the  proposal  that  DMSO  should  have  a  test-bed  using  commercial  games  to 
simulate  our  antagonists’  actions  for  training  and  rehearsal  purposes.  No  follow-up 
AFRL  action  is  recommended. 


3.2.  Guest  Lecturers 

In  an  effort  to  expose  some  of  the  latest  concepts  and  technologies  in  decision  science 
to  the  broader  R&D  community  at  AFRL,  the  DSWG  sponsored  a  number  of  guest 
lecturers  at  the  Rome  Research  Site.  A  listing  and  brief  summary  of  a  few  of  the  key 
presentations  is  provided  below. 


3.2.1.  Carmel  Domshlak,  Ph.D.,  Cornell  University;  Aug  2003. 

Title:  The  Success,  Failure,  and  Promise  of  Decision  Theory 

The  ability  to  make  decisions  and  to  assess  potential  courses  of  action  is  a  corner-stone 
of  many  artificial  intelligence  (Al)  applications,  including  expert  systems,  autonomous 
agents,  decision-support  systems,  recommender  systems,  configuration  software,  and 
constrained  optimization  applications.  To  make  good  decisions,  we  must  be  able  to 
assess  and  compare  different  alternatives,  and  thus  explicit  information  about  the 
decision-maker's  preferences  is  required. 

The  field  of  decision  theory  and  its  companion  methodology  of  decision  analysis  deal 
with  the  merits  and  making  of  decisions.  As  developed  by  philosophers,  economists, 
and  mathematicians  over  some  300  years,  these  disciplines  have  developed  many 
powerful  ideas  and  techniques,  which  exert  major  influences  over  virtually  all  the 
biological,  cognitive,  and  social  sciences.  In  spite  of  these  remarkable  achievements, 
the  tools  of  traditional  decision  theory  have  not  proven  fully  adequate  for  supporting 
recent  attempts  in  artificial  intelligence  to  automate  decision  making.  The  main  problem 
is  that  extracting  preference  information  from  users  is  generally  an  arduous  process, 
and  human  decision  analysts  have  developed  sophisticated  techniques  to  help  elicit  this 
information.  A  key  goal  in  the  study  of  computer-based  decision  support  systems  is  the 
construction  of  tools  that  allow  the  preference  elicitation  process  to  be  automated,  either 
partially  or  fully.  Methods  for  extracting,  representing,  and  reasoning  about  the 
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preferences  of  naive  users  are  particularly  important  in  artificial  intelligence  (Al) 
applications,  whose  users  cannot  be  expected  to  have  the  patience  and/or  the  ability  to 
provide  detailed  preference  relations. 

In  this  talk,  a  high-level  overview  of  the  multi-disciplinary  achievements  in  the  field  of 
decision  theory  was  presented,  and  the  speaker  addressed  the  attractiveness  and 
shortcomings  of  numerous  advancements  with  respect  to  building  automated  decision- 
support  systems.  The  tutorial  was  presented  at  a  technical  level  suitable  for  a  general 
audience  interested  in  decision  support  systems,  and  it  was  intended  to  introduce  non¬ 
specialists  to  an  Al  area  of  emerging  importance. 

Carmel  Domshlak  is  a  post-doctoral  fellow  working  with  the  Intelligent  Information 
Systems  Institute  at  Cornell  University.  His  research  focused  on  modeling  and 
reasoning  about  preferences,  as  well  as  on  exploiting  structural  properties  in 
probabilistic  reasoning,  planning,  and  some  other  areas.  His  Ph.D.  in  computer  science 
from  Ben-Gurion  University  (Israel)  was  for  his  work  on  preference  representation 
models,  and  he  has  published  numerous  scientific  papers  in  this  research  area. 


3.2.2.  Selmer  Bringsjord,  Ph.D.,  Rensselaer  Polytechnic  Institute. 

Title:  Multi-Agent  Reasoning  and  Mental  Metalogic  (MARMML)  Reasoning  System 

Professor  Bringsjord  is  the  chair  of  the  Department  of  Cognitive  Science  at  Rensselaer 
Polytechnic  Institute,  Troy,  NY.  He  is  also  a  professor  of  Computer  Science  and 
Cognitive  Science.  He  conducts  research  in  Artificial  Intelligence  as  the  director  of  the 
Rensselaer  Al  &  Reasoning  Laboratory  (RAIR),  and  has  authored  an  impressive  body 
of  work  on  the  subjects  of  Al  and  machine  reasoning.  Reference: 
http :  //www  .rpi.edu/~brings/ 

Professor  Bringsjord  presented  an  overview  multi-agent  reasoning  challenges,  their 
implementation  of  a  Multi-Agent  Reasoning  and  Mental  Metalogic  (MARMML)  system, 
and  the  ongoing  research  being  conducted  at  the  RAIR  laboratory. 


3.2.3.  Mica  Endsley,  President,  SA  Technologies. 

Title:  Designing  to  Support  Situation  Awareness 

Ms.  Endsley  presented  her  model  of  situation  awareness  (SA)  and  the  supporting 
theory  (see  Figure  2);  and  her  presentation  emphasized  practical  aspects  of  the  system 
design  process  to  improve  situation  awareness  and  avoid  “SA  Demons,”  (e.g.,  common 
pitfalls  for  false  assumptions).  Her  presentation  concluded  with  a  discussion  of 
measuring  SA,  with  emphasis  on  the  Situation  Awareness  Global  Assessment 
Technique  (SAGAT). 
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Mica  Endsley  is  a  recognized  world  leader  in  the  design,  development  and  evaluation  of 
systems  to  support  human  situation  awareness  and  decision  making,  and  serves  as 
President  of  SA  Technologies.  Prior  to  forming  SA  Technologies  she  was  a  Visiting 
Associate  Professor  at  MIT  in  the  Department  of  Aeronautics  and  Astronautics  and 
Associate  Professor  of  Industrial  Engineering  at  Texas  Tech  University. 


Figure  2.  Simplified  Endsley  Situation  Awareness  Model 


3.2.4.  CTA  Resource,  Web-based  Seminars. 

In  addition  to  those  speakers  who  personally  traveled  to  AFRL  to  lecture  on  topics  of 
interest,  the  DSWG  arranged  for  staff  members  at  AFRL  to  participate  in  web-based 
seminars  that  were  arranged  through  the  “CTA  Resource”  online  community;  see 
http://www.ctaresource.com/.  These  were  delivered  to  AFRL  at  no  cost  to  the 
Government. 

The  technology  employed  for  these  online  seminars  enabled  delivery  of  real-time  live 
audio  and  video  feeds  from  the  speaker’s  location  over  the  internet  to  AFRL,  and 
concurrently  to  many  other  sites  throughout  the  world.  A  number  of  excellent 
presentations  were  delivered;  however,  the  quality  and  reliability  of  the  audio  and  video 
feeds  was  at  times  problematic.  A  listing  of  a  few  of  the  featured  presentations  is 
provided  below: 

•  The  Empirical  Comparison  of  CTA/CWA,  Dr.  Robert  Hoffman. 

•  Field  Observation  Methods  for  Cognitive  Task  Analysis,  Dr.  Emilie  Roth. 

•  Applied  Cognitive  Work  Analysis,  Dr.  William  C.  Elm. 

•  Cognitive  Task  Analysis  for  Teams,  Dr.  Nancy  Cooke. 

Many  of  these  and  other  presentations  remain  available  via  archives  on  the  CTA 
Resource  web  site. 

3.3.  Site  Visits 

Throughout  the  lifespan  of  the  DSWG,  members  of  the  AFRL  team  made  site  visits  to  a 
number  of  government,  academic,  and  industrial  research  organizations  seeking  to 
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extend  our  knowledge  of  specific  scientific  and  technical  topics  within  the  field  of 
decision  science.  A  representative  listing  of  visits  is  provided  below: 

•  Klein  Associates;  Fairborn,  OH.  See:  http://www.decisionmaking.com/ 

•  Army  Research  Laboratory  (ARL)  Advanced  Decision  Architectures  (ADA) 
Collaborative  Technology  Alliance  (CTA);  multiple  meetings  at  various  locations. 
See:  http://www.arlada.info/. 

•  NASA  Ames  Research  Center,  Moffett  Field,  CA.  See: 
http://www.nasa.gov/centers/ames/home/index.html. 

•  University  of  Pennsylvania  (Dr.  Barry  Silverman),  Philadelphia,  PA.  See: 
http://www.seas.upenn.edu/~barryg/ 

•  Synergia,  Redwood  City,  CA.  See:  http://www.synergia.com/. 

•  Stottler-Henke,  San  Mateo,  CA.  See:  http://www.stottlerhenke.com/. 

•  Institute  for  Human  and  Machine  Cognition  (IHMC);  Pensacola,  FL.  See: 
http://www.ihmc.us/. 

3.4.  Sponsored  Research  Projects 

During  the  period  of  the  DSWG  investigation,  AFRL  sponsored  a  number  of  specific 
focused  research  projects  supporting  the  investigation  of  decision  science  and 
improving  decision  support  systems.  A  number  of  those  sponsored  research  projects 
were  guided  by  the  AFRL  DSWG  team,  and  are  summarized  in  this  section. 


3.4.1.  Implications  of  Modern  Decision  Science  and  Model 

Abstraction  Theory  for  the  Architecture  of  Decision  Support 
Systems 

The  objective  of  this  project  was  to  provide  a  critical  review  of  the  state-of-the-art  in 
relevant  decision  science,  to  extend  and  apply  research  on  model  abstraction  (and 
related  multiresolution  modeling),  and  to  draw  implications  for  the  design  and 
development  of  decision  support  systems.  The  lead  investigator  for  this  project  was 
Jonathan  Kulick,  of  RAND  Corp.  The  following  products  were  delivered: 

•  Davis,  Paul  K.,  Michael  Egner,  and  Jonathan  Kulick,  with  contributions  by  Robert 
Anderson  and  Lynne  Wainfan,  Implications  of  Modern  Decision  Science  for 
Decision  Support  Systems,  RAND  DRR-3300,  2004. 

•  Bigelow,  James  and  Paul  K.  Davis,  Implications  for  Model  Validation  of 
Multiresolution,  Multiperspective  Modeling  and  Exploratory  Analysis,  RAND,  MR- 
1570,  2003. 

•  Kulick,  Jonathan  and  Paul  K.  Davis,  Judgmental  Biases  in  Decision  Support  for 
Air  Operations,  SPIE,  2003. 

•  Davis,  Paul  K.,  Thoughts  on  Higher-Level  Adversary  Modeling,  SPIE,  2003. 

•  Davis,  Paul  K.,  Rethinking  Families  of  Models  and  Games,  SPIE,  2004. 

•  Egner,  Michael  and  Paul  K.  Davis,  Synthesis  of  Paradigms  for  Decision  Support, 
SPIE,  2004. 


19 


The  various  documents  have  many  implications  for  decision  support,  including  the  need 
to  build  in  capabilities  for  exploratory  analysis,  the  need  to  emphasize  search  for 
flexible,  adaptive,  and  robust  strategies  (rather  than  "optimal"  strategies),  new  ideas  on 
multiresolution  modeling  and  the  related  problem  of  developing  families  of  models. 


3.4.2.  An  Integrated  Approach  to  Dynamic  Decision  Making  Using 
Action  Evaluation  and  Affective  Symbology  in  Collaborative 
Environments,  University  of  Dayton 

The  goal  of  this  line  of  investigation  was  to  improve  the  efficiency,  effectiveness  and 
sustainability  of  collaborative  decision  support  systems.  Conflicts  and  a  gamut  of 
problems  are  found  when  diverse  groups  of  people  try  to  agree  on  terminology  as  they 
attempt  to  accomplish  some  shared  objective.  The  specific  objective  of  this  project  was 
to  integrate  an  Affective  Symbolic  Representation  (ASR),  derived  from  human  systems, 
with  an  Action  Evaluation  (AE)  approach  that  is  value-driven  and  a  Collaborative 
Visioning  process  in  order  to  develop  a  dynamic  decision  making  support  system.  The 
principal  investigor  was  Waleed  W.  Smari,  Ph.D.,  Electrical  and  Computer  Engineering 
Department,  University  of  Dayton. 

A  symbology  was  sought  to  represent  diverse  worldviews  of  entities  so  a  decision 
support  system  could  adapt,  on-the-fly,  to  changes  in  the  environment  and  situational 
factors.  Actual  conflict  resolution  work  was  considered  with  current  social  theory. 

Virtual  sandplay  from  psychotherapy  was  proposed  in  combination  with  an  Action 
Evaluation  collaboration  tool  to  reveal  the  complexity  of  intertwined  worldviews.  The 
central  goal  of  Action  Evaluation  is  to  find  a  joint  action  plan  that  is  acceptable  from  the 
different  worldviews  of  the  different  stakeholders  in  a  collaborative  effort,  to  facilitate  the 
discovery  of  connections  between  these  worldviews.  The  action  plan  must  be  workable 
from  the  perspective  of  all  stakeholders.  Tailorability  is  proposed  to  facilitate  adaptation 
to  user  needs. 

At  the  conclusion  of  the  DSWG  reporting  period  this  research  was  ongoing,  under  the 
direction  of  Dr.  Kirk  Weigand,  AFRL/IFSD,  (Ref.  Contract#  F30602-03-2-0126). 


3.4.3.  Efficient  Campaign  Plan  Selection  and  Evaluation 

A  challenging  research  problem  in  the  planning  of  military  effects-based  operations 
(EBO)  is  the  efficient  selection  of  action  plans.  Existing  effects-based  planning  and 
analysis  tools  such  as  AFRL’s  Causal  Analysis  Tool  (CAT)  offer  great  promise  to  help 
planners  model  and  analyze  an  effects-based  plan,  but  offer  little  help  to  planners 
seeking  an  optimal  or  near-optimal  plan  solution.  This  is  a  technology  gap  that  could 
limit  the  utility  of  tools  such  as  CAT,  and  ultimately  the  development  of  EBO  concepts. 
Under  this  project  Dr.  Ji  investigated  two  methods  to  address  this  problem:  one  was  a 
graph-theoretic  approach  and  the  other  was  an  “action-ensemble  based  greedy 
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algorithm”  approach.  The  graphical  approach  was  aimed  at  developing  a  theoretical 
basis  to  reduce  the  search  space  and  to  more  methodically  search  for  the  optimal  or  a 
near-optimal  plan  in  an  efficient  manner.  The  action  ensemble  based  greedy  approach 
recursively  identifies  an  ensemble  of  actions  to  maximize  the  performance  of  an 
ensemble  in  achieving  the  goal.  Preliminary  empirical  studies  indicate  both  methods 
have  promise  in  producing  optimal  or  near-optimal  solutions  in  reasonably  bounded 
periods  of  time. 

The  principal  investagor  was  Qiang  Ji,  Ph.D.,  Assistant  Professor;  Department  of 
Electrical,  Computer,  and  System  Engineering;  Rensselaer  Polytechnic  Institute.  His 
research  was  conducted  at  AFRL  during  the  summer  of  2003,  under  a  visiting  faculty 
research  program  sponsored  through  the  AFRL  Information  Institute.  Professor  Ji’s 
time  working  at  AFRL  has  lead  to  follow-on  research  in  which  he  is  applying  dynamic 
bayes  network  concepts  from  this  campaign  planning  research  to  challenges  in  dynamic 
information  fusion  under  uncertainty. 


3.4.4.  Supporting  Team  Decision  Processes  in  Adversarial  Task 
Environments:  A  Multi-Disciplinary  Investigation  into  Assessing 
the  Functionality  of  Information  Technologies 

This  research  effort  focused  on  multi-disciplinary  theory  development  and  hypotheses 
generation  on  the  functionality  of  information  technologies  for  team  decision  support  in 
hyper-competitive  environments1.  A  multi-disciplinary  research  team  of  graduate  (and 
exceptional  undergraduate)  students,  under  the  direction  of  the  principal  investigator 
identified  the  theoretic  underpinnings  of  team-based  decision  systems,  and  then 
examined  the  theory  via  a  priori  assessment  of  AFRL  tools  for  decision  support.  The 
principal  investigator  was  Thomas  Triscari,  Jr.  Ph.D.,  Lally  School  of  Management  and 
Technology,  Rensselaer  Polytechnic  Institute. 

Background:  To  gain  a  strategic  advantage  over  an  intelligent  adversary  in  hyper- 
competitive  environments,  military  commanders  or  business  executives  are  looking 
toward  information  and  computational  technology  to  help  anticipate  the  effects  of  a  set 
of  inter-related  decisions  (actions)  and  counter-decisions.  For  many  of  these  decisions, 
the  underlying  processes  are  team-based.  That  is,  information  (which  is  often  uncertain 
and  equivocal)  is  somehow  aggregated  from  a  set  of  individuals  within  the  “command 
staff”  (or  senior  management)  performing  tasks  which  may  be  supported  by  information 
and  computational  technology.  Despite  the  inherent  collaborative  nature  of  the 
processes,  many  information  systems  are  focused  on  supporting  individual  tasks  and 
presume  the  linking  of  individual  decision  aids  will  provide  effective  team  support. 
Anecdotal  evidence  suggests  that  “information  systems”  designed  without  an 
understanding  of  human  behavior  in  stressful  situations  can  have  dysfunctional  effects 
on  decision  making  performance.  This  suggests  a  multi-disciplinary  (e.g.,  decision 


1  Hyper-Competitive  Environments  include  those  in  which  decisions  must  be  made  with  incomplete,  uncertain 
information;  ambiguous  goals;  high  stakes,  and  time-constrained,  and  against  intelligent  adversary(ies). 
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sciences)  approach  is  needed  to  support  team  processes.  That  is,  information 
technology  must  be  employed  in  a  deliberate,  holistic  way  to  support  the  information¬ 
intensive,  team  processes  needed  to  achieve  competitive  advantage  (e.g.,  information 
superiority)  over  an  intelligent  adversary. 

Results:  After  conducting  a  thorough  review  of  the  scientific  literature,  and  interacting 
extensively  with  the  AFRL  research  staff  over  the  course  of  the  summer  2003,  the 
Rensselaer  team  presented  a  summary  of  their  analysis  and  a  set  of  recommendations: 

(A)  Multi-disciplinary  investigations  are  needed  to  explore  the  field  of  decision  science 

and  decision  support  systems  (DSS)  holistically. 

(1 )  To  achieve  effective  evaluation  of  DSS  tools  the  system  must  be  tested  using 
experiments  in  a  naturalistic  laboratory  environment. 

(2)  Experiments  must  seek  to  understand  how  human  behavior  and  cognition  effect 
system  performance. 

(3)  Experiments  must  seek  to  understand  what  impact  the  machine  would  have  on  a 
team  or  command  staff. 

(4)  A  multidisciplinary  team  is  required  to  design,  under  take,  and  carry  out  such 
experiments. 

(B) A  naturalistic  adversarial  task  environment  is  needed  to  objectively  advance  our 

understanding  of  team  decision  making  in  hyper-competitive  environments: 

(C) Key  research  issues  that  should  be  addressed: 

(1 )  How  do  we  develop  and  evaluate  decision  technologies  within  the  expanded 
context  of  decision  systems? 

(2)  How  can  a  multidisciplinary  team  be  employed  to  construct  decision  systems  that 
will  be  effective  in  a  hypercompetitive  environment? 

(3)  How  can  the  scientific  research  methods  be  used  to  design,  develop,  and 
evaluate  systems  from  a  user  centered,  human  abilities  driven  approach? 

(4)  What  methods  should  be  used  in  the  design/assessment  of  decision  systems  to 
reduce  adverse  consequences? 

(5)  What  organizational  structure,  values,  and  initiative  will  best  facilitate  innovative 
decision  technology  design? 

A  select  set  of  presentation  charts  from  the  Rensselaer  team’s  final  briefing  is  provided 
in  Appendix  H. 
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4.0  Summary  of  Accomplishments 

In  this  section  the  major  accomplishments  of  the  DSWG  are  presented,  followed  by  a 
discussion  of  the  areas  in  which  the  work  group  fell  short  of  fully  achieving  the  expected 
end-state. 

4.1.  Major  Accomplishments 

A  brief  summary  of  the  most  significant  accomplishments  resulting  from  the  activities  of 
the  DSWG  are  presented  below. 

Respected  authors  and  leading  researchers  in  the  decision  science  field  participated  in 
the  workshops,  or  accepted  invitations  to  lecture  at  AFRL  sponsored  events.  As  a 
result,  significant  numbers  of  the  AFRL  S&T  staff  were  exposed  to  leading  theories  and 
research  in  decision  making  and  decision-support  technologies,  and  were  given  an 
opportunity  to  develop  a  network  with  world  leading  experts.  While  anecdotal,  there  is 
already  evidence  that  this  exposure  has  resulted  in  the  internalization  of  new  concepts 
and  incremental  improvements  in  our  practices. 

The  majority  of  these  DSWG  activities  were  conducted  cooperatively  between  the 
Information  (IF)  and  Human  Effectiveness  (HE)  directorates  of  AFRL.  As  a  result,  the 
degree  of  interaction  between  the  two  directorates  has  been  significantly  elevated, 
yielding  greater  understanding  and  the  identification  of  numerous  opportunities  for 
future  collaboration  in  the  development  of  decision  support  systems  and  underpinning 
technologies. 

The  activities  of  the  DSWG  contributed  directly  to  the  development  of  a  RAND  Report 
entitled  “Implications  of  Modern  Decision  Science  for  Military  Decision-Support 
Systems,”  Project  Air  Force,  2005.  That  monograph  selectively  reviews  a  number  of 
key  topics  of  relevance  and  interest  to  the  development  of  decision  support-systems, 
and  makes  a  significant  contribution  both  to  our  understanding  of  the  field  and  in 
identifying  opportunities  to  advance  the  state-of-practice  in  DSS  development. 

A  number  of  specific  research  projects  have  emerged  from  the  work  group  activities, 
each  in  its  own  right  extending  our  understanding  of  decision  science  and/or  making 
advances  in  key  underpinning  technologies  of  decision  support  systems: 

•  Implications  of  Modern  Decision  Science  and  Model  Abstraction  Theory  for  the 
Architecture  of  Decision  Support  Systems 

•  An  Integrated  Approach  to  Dynamic  Decision  Making  Using  Action  Evaluation 
and  Affective  Symbology  in  Collaborative  Environments 

•  Efficient  Campaign  Plan  Evaluation  and  Selection 

•  Supporting  Team  Decision  Processes  in  Adversarial  Task  Environments 
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4.2.  DSWG  Shortfalls 

A  brief  discussion  of  the  areas  where  the  DSWG  fell  short  of  fully  realizing  the 
objectives  is  presented  below. 


4.2.1.  Distinguished  Lecture  Series 

Although  a  number  of  world-class  speakers  were  brought  in  to  lecture  at  AFRL  under 
the  sponsorship  of  the  DSWG  and  those  sessions  were  very  well  attended,  due  to 
scheduling  difficulties  and  conflicts  the  number  of  presentations  that  were  held  fell  short 
of  our  initial  goals  and  expectations.  As  a  result,  the  breadth  of  topics  that  were 
presented  was  somewhat  limited. 


4.2.2.  Evaluating  the  State-of-the-Art 

In  an  effort  to  better  understand  the  state-of-the-art  in  decision  science,  one  of  the  tasks 
the  workgroup  initiated  was  to  develop  a  well  organized  taxonomy  of  the  sciences  and 
technologies  that  fall  within  the  scope  of  “decision  science.”  The  breadth  of  the  topic, 
and  the  many  different  views  of  the  space  of  decision  science  made  this  a  more 
challenging  task  than  was  anticipated,  and  was  never  brought  to  closure.  Additionally, 
the  breadth  of  disciplines  within  the  decision  science  field  prevented  the  work  group 
from  reaching  a  final  assessment  of  the  states-of-th e-art  across  the  many  contributing 
fields  of  science  and  technology.  The  team;  however,  did  gain  a  more  insightful 
understanding  of  and  appreciation  for  the  state-of-practice  in  the  development  of 
decision  support  systems,  and  identified  a  number  limitations  and  areas  of  concern. 


4.2.3.  Computational  Technologies  in  Decision  Support  Systems 

The  workgroup  put  a  great  deal  of  emphasis  into  the  cognitive  science  aspects  of 
decision  science,  and  with  limited  resources,  this  came  at  the  expense  of  a  deeper 
exploration  of  state-of-the-art  and  current  limitations  in  computational  technologies  and 
their  application  in  decision-support  systems.  As  a  result,  the  workgroup  did  not 
investigate  the  role  of  computational  technologies  in  decision  science  as  broadly  or  as 
deeply  as  may  be  warranted. 


4.2.4.  Institutionalizing  the  Process 

The  workgroup  made  great  strides  in  bringing  together  the  personnel  and  diverse 
cultures  of  two  different  AFRL  directorates  to  create  an  organization  that  is  more 
capable  of  delivering  world  class  decision-support  systems  for  the  Air  Force.  However, 
these  gains  were  not  institutionalized  within  the  organization  via  any  formal  means  (e.g., 
changes  in  processes).  The  effort  of  the  workgroup  also  seems  to  have  “infected”  a 
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broader  community  within  AFRL  with  an  appreciation  for  the  potential  payoffs  of 
establishing  “joint”  programs  across  disciplines  and  directorates.  However,  this  course 
has  not  yet  been  fully  internalized  and  will  likely  require  follow-up  action  to  sustain  the 
momentum  achieved  by  the  DSWG. 


5.0  Conclusion  &  Recommendations 

Although  all  the  objectives  set  forth  for  the  DSWG  late  in  2002  were  not  fully  achieved, 
the  efforts  of  the  working  group  have  paid  off  in  a  number  of  important  ways: 

•  Improving  the  frequency  and  quality  of  the  interactions  across  the  two 
participating  AFRL  directorates  in  matters  related  to  decision  science. 

•  Highlighting  the  need  for  a  multi-disciplinary  perspective  and  cooperation  in  the 
advancement  of  decision  science,  and  in  the  development  of  decision  support 
systems. 

•  Exposing  AFRL  scientists  and  engineers  to  many  of  the  leading  experts  and 
widely  accepted  scientific  theories  that  form  the  foundation  of  our  understanding 
of  decision-making  and  decision  support  systems. 

•  Expanding  the  portfolio  of  decision  science  research  at  AFRL  in  directions  that 
are  critical  to  meet  future  Air  Force  decision-support  needs,  and  doing  so  in  ways 
that  build  smartly  on  an  extensive  established  multi-disciplinary  research  base. 

Recognizing  that  the  DSWG  did  not  achieve  all  of  the  objectives  envisioned,  the 
following  recommendations  for  further  action  are  offered: 

•  The  DSWG  served  an  important  role  in  building  and  strengthening  ties  between 
two  geographically  separated  communities  of  AFRL  scientists  and  engineers 
working  in  the  field  of  decision  science,  and  while  the  ties  have  strengthened, 
these  gains  could  steadily  evaporate  without  persistent  tending.  Both 
directorates  should  seek  ways  to  maintain  and  further  increase  the  momentum 
gained  by  the  DSWG,  and  consider  expanding  the  membership  or  collaboration 
beyond  that  of  AFRL: 

o  Charter  a  DSWG-like  working  group  on  a  long-term  basis,  and  allocate 
resources  annually  (manpower  and  funding);  to  guide,  coordinate,  and 
report  on  AFRL’s  progress  in  decision  science.  Within  AFRL,  reach  out 
beyond  the  two  divisions  (IFS  and  HEC)  which  formed  the  core  of  the 
DSWG. 

o  Sponsor  research  projects,  that  would  require  principal  investigators  or 
project  managers  from  the  two  respective  directorates  (HE  and  IF)  to 
jointly  participate  or  manage  the  investigation. 

o  Work  collaboratively  with  AFOSR  to  jointly  sponsor  and  direct  research 
topics  through  mechanisms  such  as  Broad  Agency  Announcements 
(BAAs). 
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o  Promote  short-term  (e.g.,  2-6  months),  temporary  duty  assignments 
related  to  decision  science  research  and  development,  to  encourage 
researchers  to  broaden  their  experience  by  working  in  a  different  AFRL 
directorate. 

o  Establish  a  joint-service  working  group  in  decision  science,  not  unlike  the 
Decision  Aids  Working  Group  (DAWG),  which  served  as  a  decision  aids 
science  and  technology  coordinating  body  for  OSD  through  the  Reliance 
process  over  a  decade  ago.  Clearly  there  is  much  to  be  gained  by 
learning  from  and  better  coordinating  with  DS  researchers  within  the  other 
services  and  government  agencies. 

The  activities  sponsored  by  the  DSWG  to  increase  AFRL  staffs  awareness  and 
understanding  of  decision  science  research  were  very  well  received.  With  or 
without  a  DSWG,  these  types  of  activities  should  be  continued  and  encouraged: 
o  Organizing  joint  workshops  on  focused  DS-related  topics  of  interest  for 
researchers  within  both  directorates  and  AFOSR,  placing  an  emphasis  on 
focus,  workshop  products,  and  follow-through, 
o  Hosting  a  DS  expert  lecturer  series  at  AFRL  facilities,  and  providing 
access  for  remote  participation  via  AFRL’s  corporate  video¬ 
teleconferencing  technology. 

o  Conducting  an  aggressive  campaign  of  site  visits  to  universities, 
government  agencies,  and  commercial  organizations  to  maintain 
awareness  of  their  respective  research  investments  and  expertise  in  DS; 
placing  particular  emphasis  on  collecting,  organizing  and  sharing  across 
AFRL  all  of  the  information  collected  during  each  site  visit. 

Finally,  with  respect  to  the  AFRL  DS  research  agenda,  a  number  of  topics  with 
high  potential  payoff  were  considered  by  the  DSWG.  The  following  brief 
consolidated  list  of  topics  is  offered  for  consideration: 

o  Mixed-initiative  decision  making:  a  holistic  multi-disciplinary  approach 

■  Approaches  and  technologies  to  minimize  the  “out  of  loop 
syndrome” 

■  In  the  context  of  distributed  teams 

■  Progressing  from  novice  to  expert 

■  Supported  by  enduring  personalized  assistants/agents 

o  Novel  approaches  for  the  empirical  assessment  of  decision  support 

■  Human-centered  engineering  vice  preference  management 

■  Quantitatively  measuring  decision-making  performance  in  teams 
o  Team  decision  making  and  collaboration:  team  /  shared  situation 

awareness  (across  perception,  cognition  and  projection), 
o  Formalizations  and  frameworks  for  identifying  the  ‘value  of  information’ 
and  dealing  with  information  uncertainty 
o  Cultural  effects  and  human  behavior  modeling:  individuals,  small  groups, 
and  societies 

o  Beyond  NP-hard:  ways  to  better  quantify  or  qualify  problem  complexity  in 
decision  making  and  computability. 


o  Unmanned  Aerial  Vehicle  (UAV)  situation  awareness  across  teams. 
(Note:  this  application-specific  topic  is  expected  to  be  of  high-interest  in 
the  coming  years  and  is  the  subject  of  an  Air  Force  SAB  study.) 
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Appendix  F.  CSE  Seminar  Presentation  Summaries 

5-7  Aug  2003 

This  appendix  is  organized  in  accordance  with  the  schedule  of  the  seminar.  For 
each  topic  presented  there  is  a  brief  statement  about  the  speaker  and  then  a 
sub-paragraph  containing  a  synopsis  of  their  presentation(s). 

1.  Robert  Hoffman,  University  of  West  Florida  (UWF) 

Robert  R.  Hoffman,  Ph.D.,  a  Research  Scientist  at  the  Institute  for  Human  and 
Machine  Cognition  (IHMC)  of  the  University  of  West  Florida,  started  the 
conference  by  presenting  an  overview  of  CSE  and  followed  that  with  a  specific 
presentation  on  a  Knowledge  Elicitation  (KE)  method  called  “concept  mapping”. 

CSE  Overview 

CSE  seeks  to  address  the  challenge  of  preserving  wisdom  and  revealing  the 
knowledge  and  skills  of  experts.  New  technologies  make  it  not  only  possible  but 
relatively  easy  to  preserve  wisdom.  We  know  how  to  elicit  the  knowledge  of 
experts,  including  “tacit”  and  “intuitive”  knowledge  that  has  gone  undocumented. 
We  know  how  to  preserve  corporate  and  organizational  memory.  We  know  how 
to  preserve  knowledge  in  a  way  that  makes  it  easy  to  be  shared  and 
disseminated.  The  plethora  of  methods  and  tools  that  abound  may  be  identified 
under  the  following  groups  although  not  limited  to  this  categorization: 
System-oriented  methods 
Cognitive-oriented  methods 

Cognitive  Work  Analysis  (CWA) 

Cognitive  Task  Analysis  (CTA) 

Cognitive  Functional  Analysis  (CWA) 

Interviewing/observing  methods  for  knowledge  elicitation 

Process  tracking  methods 

Conceptual  methods 

Computational  cognitive  modeling 

System  evaluation 

Descriptive  approaches 

Human  reliability  analysis 

There  is  no  universally  accepted  taxonomy  for  CSE.  Some  experts  see  CSE  as 
distinct  from  Cognitive  Work  Analysis  (CWA)  and  Cognitive  Task  Analysis  (CTA), 
while  others  view  those  methods  as  subsets  of  CSE. 
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Concept  Mapping 

Dr.  Hoffman  presented  concept  mapping  as  a  tool/technique  for 
capturing/preserving  knowledge  within  a  domain.  Concept  Mapping  is  based  on  a 
theory  of  knowledge,  a  theory  of  learning,  research  in  education  and  research  in 
diagrammatic  reasoning.  Concept  Maps  (Cmaps)  represent  knowledge  in 
diagrams  that  express  concepts  and  propositions  where  a  proposition  is  the 
expression  of  the  relationship  between  concepts.  Cmaps  are  typically  generated 
in  conjunction  with  a  specialized  interview  process,  can  be  applied  to  almost  any 
domain,  and  reflect  various  levels  of  sophistication  (organization  of  information). 
Dr.  Hoffman  characterized  the  interview  process  that  occurs  and  then  showed 
how  concept  maps  evolved  and  were  actually  used  as  part  of  the  interface  for 
systems  in  weather  forecasting,  and  radar  tracking.  Following  his  general 
presentation  he  demonstrated  how  the  Cmap  tool,  which  is  downloadable  from 
the  IHMC  web  site,  can  be  used  to  facilitate  the  interview  process  and  the 
capture  of  “expert”  knowledge.  Participants  with  laptops  installed  the  Cmap  tool 
from  the  Resource  Kit  and  started  working  with  it  under  Dr.  Hoffman’s  guidance. 

2.  Isaac  Brewer,  Pennsylvania  State  University  (PSU) 

Isaac  Brewer  is  working  at  the  GeoVISTA  Center  at  Penn  State  University,  where 
he  is  finishing  his  Ph.D.  under  the  direction  of  Dr.  MacEachren.  He  is  focused  on 
developing  a  Cognitive  Systems  Engineering  approach  to  guide  design  of 
advanced  interfaces  that  support  collaborative  interaction  with  a  Geographical 
Information  System  (GIS)  in  emergency  management  situations. 

A  Four-Stage  Approach  to  Knowledge  Elicitation 

He  described/advocated  a  four-stage  knowledge  elicitation  process  that 
progresses  from  a  simple  introductory  task  analysis  all  the  way  through  to  actual 
event  observation  and  verification  of  results.  The  stages  were  defined  as: 

1 .  Initial  Task  Analysis  -  to  determine  where  to  focus  the  effort 

2.  Work  Domain  Analysis  -  to  collect  artifacts  and  information  from  experts 
at  their  work  location  to  build  the  Knowledge  Engineer’s  understanding  of 
the  domain  of  practice 

3.  Scenario  Creation  and  Design  Storyboarding  -  to  model  the  domain 
process  and  provide  a  proposed  hypotheses  for  the  design  of  the 
prototype  system 

4.  Event  Observation  (Optional)  -  to  allow  the  knowledge  engineer  to  verify 
the  results  collected  from  the  knowledge  elicitation  exercises  of  the 
previous  three  stages,  to  identify  inconsistencies,  and  to  witness  those 
special  circumstances  in  which  the  system  potentially  breaks  down  that 
only  arise  during  the  actual  events  themselves. 

The  seminar  participants  were  divided  into  seven  groups  and  given  a  concept 
map  (developed  while  implementing  stage  2  of  his  process)  describing  the 
operational  scenario  (from  various  perspectives)  that  occurs  in  emergency 
management  preparation  to  respond  to  a  hurricane.  Participants  were  instructed 
to  “study”  the  concept  map  and  “figure  out”  how  to  use  it  to  assist  in  designing  a 
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prototype  system.  The  sheer  size  of  the  map  (5’  square  with  hundreds  of  nodes 
and  propositions  and  4  distinct  perspectives)  emphasized  the  complexity  of 
knowledge  represented  on  the  map,  and  gave  participants  a  sense  of  the  reality 
of  capturing  knowledge. 


3.  Gary  Klein,  Klein  Associates,  Inc. 

Dr.  Klein,  founder  and  Chief  Scientist  of  Klein  Associates,  has  devoted  his  career 
to  studying  how  people  make  decisions  in  natural  settings  and  under  pressure 
and  uncertainty.  His  presentation  addresses  decision-centered  design,  where 
knowledge  of  the  key  decisions  to  be  made  is  what  drives  the  design. 

Decision-Centered  Design  (DCD) 

In  his  presentation  on  DCD  Dr.  Klein  addressed  the  following  questions: 

•  What  Cognitive  Functions  are  we  supporting? 

•  What  is  the  rationale  for  a  DCD  approach? 

•  What  is  the  process  for  doing  DCD? 

•  What  are  some  examples  of  DCD  projects? 

•  How  does  DCD  frame  a  Cognitive  Task  Analysis? 

He  described  a  collection  of  actions/activities  (cognitive  functions)  that  constitute 
macro  cognition  and  further  characterized  macro  cognition  as  a  framework  for 
carrying  out  cognitive  systems  engineering. 

He  defined  DCD  as  a  Cognitive  Systems  Engineering  approach  for  developing  IT 
solutions,  wherein  the  design  drivers  are  the  key  decisions  users  have  to  make. 
Part  of  the  rationale  for  DCD  is  that  critical  decisions  appear  to  offer  more 
leverage  than  other  aspects  of  cognition.  Decision  requirements  are  enduring 
aspects  of  the  mission  while  tasks  and  equipment  may  change.  He  indicated 
that  DCD  is  needed  to  address  information  overload,  and  the  “disconnect”  that 
often  occurs  between  users  and  designers.  DCD  is  most  useful  when  strong 
cognitive  activity  and  expertise  are  important  for  task  success.  He  described  the 
DCD  process  with  a  flow  diagram  that  included  several  activities  that  used  other 
CSE  related  methods  such  as  developing  decision  requirements  tables  and 
implementing  cognitive  task  analysis  techniques.  He  discussed  the  significant 
role  of  metrics  development  as  part  of  the  DCD  process  with  emphasis  on 
performance  improvement.  Cognitive  function  metrics  are  generated  during 
conceptual  design  as  V&V  criteria.  Macro  cognitive  functions  are  used  to  shape 
the  metrics  (problem  detection,  sense  making,  re-planning,  coordination). 
Cognitive  Task  Analysis  (CTA)  data  is  used  to  generate  cognitive  metrics. 

Dr.  Klein  then  proceeded  to  discuss  CTA  in  detail.  He  defined  CTA  as  the 
application  of  a  family  of  methods  and  tools  for  gaining  access  to  the  mental 
processes  that  organize  and  give  meaning  to  observable  behavior.  CTA  results 
describe  the  cognitive  processes  that  underlie  performance  of  tasks,  and  the 
cognitive  skills  needed  to  perform.  Expertise  is  the  central  focus  of  CTA.  CTA  is 
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primarily  valuable  for  tasks  that  depend  on  cognitive  aspects  of  expertise,  such 
as  decision  making  and  problem  solving.  CTA  is  not: 

•  Research  into  the  nature  of  basic  cognitive  processes 

•  A  prescription  about  how  people  should  be  thinking 

•  A  behavioral  task  analysis 

Throughout  his  talk  Dr.  Klein  kept  the  participants  involved  by  asking  them 
questions  about  their  work,  and  the  relevance  of  his  statements. 

4.  Mica  R.  Endsley,  SA  Technologies,  Inc. 

Mica  Endsley  is  a  recognized  world  leader  in  the  design,  development  and 
evaluation  of  systems  to  support  human  situation  awareness  and  decision 
making,  and  serves  as  President  of  SA  Technologies.  Her  chosen  topic  for  the 
seminar  addresses  designing  for  Situation  Awareness  in  complex  systems. 

Designing  to  Support  Situation  Awareness 

Dr.  Endsley  defines  Situation  Awareness  (SA),  the  driver  of  the  decision  process 
and  a  key  factor  in  determining  decision  quality,  as  “the  perception  of  the 
elements  in  the  environment  within  a  volume  of  time  and  space,  the 
comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near 
future”.  SA  is  also  the  product  of  processes  (perception,  attention,  synthesis  & 
analysis,  pattern  matching  with  long  term  memory)  that  contribute  to  memory. 

She  talked  about  how  we  form  mental  models  and  described  how  novices 
develop  SA  versus  experts.  Novices  and  novel  situations  use  data  driven 
processing,  while  experts  use  a  goal-driven  process  to  develop  SA.  She  then 
proceeded  to  describe  “SA  Demons”  and  discuss  how  they  influence  the  level  of 
SA  for  complex  systems.  Some  of  the  demons  are: 

•  Attentional  tunneling 

•  Requisite  Memory  Trap 

•  WAFOS:  Workload,  Fatigue,  &  Other  Stressors 

•  Data  Overload 

•  Misplaced  Salience 

•  Complexity  Creep 

•  Errant  Mental  Models 

•  Out-of-the-Loop  Syndrome 

She  described  how  we  “fight”  the  SA  demons  when  designing  complex  systems 
by  (1)  specifically  identifying  requirements  for  SA,  (2)  considering  human  factors 
design  guidelines  and  standards  as  we  design,  and  (3)  incorporating  test  and 
evaluation  procedures  to  measure  SA  into  our  design.  She  advocates  using 
“goal-directed  task  analysis  (GDTA)”  to  establish  the  SA  requirements,  but 
cautions  that  a  difficult  element  of  GDTA  is  distinguishing  goals  from  physical 
tasks  and  information  needs.  She  advocates  for  human-centered  design  over 
technology-centered  design  since  it  focuses  on  designing  the  technology  to  fit  the 
capability  of  humans,  not  vice  versa.  She  then  went  into  detail  about  design 
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principles  for  SA-oriented  design.  Some  key  points  related  to  these  principles 
address  how  uncertainty  affects  SA  and  decision  making.  They  are: 

•  How  information  is  presented  is  critical 

•  Dissonant  information  has  a  greater  impact  on  decisions  than  missing 
data 

•  Format  for  presenting  reliability  or  confidence  matters 

Her  discussion  of  design  principles  relating  to  alarms  revealed  that  although 
alarms  are  theoretically  considered  to  aid  SA,  in  reality,  they  often  fail  to  aid  SA, 
because  there  are  too  many,  or  there  are  high  false  alarm  rates,  or  they  have 
been  disabled  because  they  are  a  nuisance  in  normal  operational  scenarios. 

SA-oriented  design  provides  a  systematic  methodology  and  tools  for  enhancing 
SA  in  designing  tomorrow’s  systems.  Measurement  of  SA  in  evaluation  system 
design  options  provides  critical  information  with  greater  sensitivity  than  simple 
performance  measures  because  it  provides  a  measure  of  the  operator’s  ability  to 
dynamically  integrate  multiple  pieces  of  information  into  a  coherent  picture  under 
operational  challenges.  This  is  critical  for  successful  performance.  One 
measurement  technique  described  by  Dr.  Endsley  is  called  the  “Situation 
Awareness  Global  Assessment  Technique  (SAGAT)”.  It  is  comprised  of  three 
main  activities: 

•  At  random  times,  freeze  the  exercise 

•  Administer  a  rapid  battery  of  queries  to  ascertain  SA  at  that  point  in  time 

•  Score  on  the  basis  of  objective  data  derived  from  the  simulation/exercise. 

SAGAT  overcomes  problems  for  collecting  SA  data  after  the  fact,  and  is  a  direct, 
unbiased,  objective  measure  of  SA,  which  is  heavily  validated,  but  it  does  require 
interruption  of  the  exercise/simulation.  Dr.  Endsley  gave  some  examples  of  how 
SA  measures  are  represented  in  a  quantifiable  format  and  the  data  used  to 
modify  the  design  concept. 

5.  Nancy  Cooke,  Arizona  State  University  (ASU) 

Nancy  Cooke  was  accompanied  by  Preston  Kiekel,  one  of  her  doctoral  students. 
They  have  been  working  together  in  recent  research  and  shared  the  presentation 
time  addressing  different  aspects  of  team  cognition.  Dr.  Cooke  provided  an 
overview  of  team  cognition  and  then  presented  the  Pathfinder  Network  Scaling 
technique.  Mr.  Kiekel  followed  her  with  a  presentation  on  the  ProNet  technique 
for  applying  network  scaling  to  sequential  data. 

Knowledge  Elicitation  Meets  Team  Cognition 

Dr.  Cooke  presented  an  overview  of  Team  Cognition  that  addressed  the 
following  questions: 

•  How  do  we  get  from  individual  knowledge  to  team  cognition? 

•  What  is  “Team  Cognition”? 

•  What  are  the  challenges  to  assessing  Team  Cognition? 
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•  What  are  some  approaches  to  Assessment? 

A  team  is  defined  as  “...a  distinguishable  set  of  two  or  more  people  who  interact 
dynamically,  interdependently,  and  adaptively  toward  a  common  and  valued 
goal/object/mission,  who  have  each  been  assigned  specific  roles  or  functions  to 
perform,  and  who  have  a  limited  life  span  of  membership”. 

She  notes  that  cognitive  engineering  directed  at  teams  is  a  relatively  new  field  of 
study  which  began  in  1996.  Assessment  of  team  cognition  addresses  team  SA 
and  “shared  mental  models”. 

Team  cognition  is  more  than  the  sum  of  the  cognition  of  individual  team 
members;  it  emerges  from  the  interplay  of  the  individual  cognition  of  each  team 
member  and  team  process  behaviors. 

Team  cognition  is  assumed  to  contribute  to  team  performance — now  more  than 
ever.  Understanding  the  team  cognition  behind  team  performance  should 
facilitate  interventions  (design,  training,  selection)  to  improve  performance.  Some 
key  challenges  to  assessing  team  cognition  are: 

•  Assumption  of  Homogeneity  -  Teams  are,  by  definition,  heterogeneous. 
This  complicates  “shared”  knowledge.  Shared  knowledge  encompasses 
both  common  and  complementary  knowledge,  and  shared  perspectives. 

•  Elicitation  is  limited  to  the  individual  level 

•  Assessment  is  valuable  to  the  extent  that  it  is  conducted  in  real-time  and 
embedded  in  task  execution  (requires  automated  measures),  or  better 
yet,  prior  to  task  execution  (based  on  performance  precursors) 


Approaches  to  Assessments 

Two  types  of  metrics  are  typically  used  in  assessments:  (1 )  Holistic  metrics  that 
elicit  at  the  team  level  (e.g.  consensus  assessment  tasks),  and  (2)  Collective 
metrics  that  elicit  at  the  individual  and  aggregate  to  estimate  the  team  level. 
Experiments  at  their  laboratory  have  resulted  in  the  following  findings: 

•  Collective  measures  predict  team  performance  for  co-located  teams 
better  than  holistic  metrics 

•  Holistic  metrics  based  on  the  consensus  tasks  predict  performance  better 
for  distributed  teams. 

•  Consensus  measures  correlate  moderately  with  performance  compared  to 
collective  measures. 

Communication  Analysis  is  a  response  to  holistic  assessment  for  operational 
environments.  Dr.  Cooke  views  this  as  a  solution  because  (1 )  it  is  real  time, 
embedded  in  the  task,  (2)  it  is  observable,  (3)  it  reflects  team  cognition  at  the 
holistic  level,  and  (4)  it  is  rich  and  multidimensional.  However  it  requires  labor 
intensive  transcription  and  coding. 
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In  summary  the  key  points  to  remember  about  team  cognition  are: 

•  Understanding  team  cognition  is  critical  for  diagnosis  of  team  dysfunction 
or  excellence  and  later  intervention  with  training,  technology,  or  team 
composition. 

•  In  operational  environments  diagnosis  needs  to  be  real-time,  task- 
embedded,  predictive,  and  automatic. 

•  Need  to  move  toward  diagnosis  by  connecting  clusters  of  symptoms  to 
diagnosis  of  team  dysfunction  or  excellence. 

Pathfinder  Network  Scaling:  Representing  Knowledge  for  Individuals  and 
Teams 

Pathfinder  is  a  conceptual  KE  method  that  takes  as  input  pair  wise  estimates  of 
relatedness  (i.e.,  proximities)  and  generates  network  representations  of  nodes 
and  links.  Concepts  are  represented  as  nodes  and  conceptual  relations  as  links 
between  nodes.  The  idea  is  to  reduce  the  estimates  in  a  psychologically 
meaningful  way.  It  thus  seeks  to  address  the  “goodness”  of  the  knowledge.  It  is 
predicated  on  semantic  network  theory  which  assumes  that  semantics  or 
meaning  is  represented  symbolically  as  a  network  of  interconnected  concepts. 
The  concept  set  and  referent  structures  are  critical  to  the  success  of  this  method. 
Pathfinder  can  be  used  to: 

•  Elicit  Knowledge  of  Experts  and  Novices 

•  Investigate  Subtle  Knowledge  Differences  Among  Experts 

•  Assess  Student  Knowledge 

Dr.  Cooke  demonstrated  a  tool  called  KNOT  which  implements  the  pathfinder 
technique. 

Applying  ProNet  to  Communication  Analysis 

ProNet,  which  stands  for  “Procedural  Networks”  is  a  technique  for  eliciting 
knowledge  by  tracking  and  analyzing  sequences  of  behaviors  or  events. 

The  technique  is  useful  for  tracking: 

•  Communication  turn-taking  sequences  to  assess  information  flow 
strategies 

•  Individual  task  sequences  of  experts  and  novices,  by  comparison 

•  Task  sequences  associated  with  high  productivity 

•  Procedural  knowledge  that  experts  cannot  express  directly 

Key  factors  that  motivate  use  of  this  technique  are: 

•  Elicitation  of  procedural  knowledge 

•  Automatic  (after  nodes  defined) 

•  Descriptive,  and  hence  robust 

•  Straightforward  data  reduction 

•  Possible  graphical  depiction 
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It  attempts  to  identify  sequences  of  prominence  and  does  not  require  lots  of  data. 
Preston  conducted  a  wonderful  “hands-on”  exercise  to  communicate  the  ProNet 
technique.  Participants  were  divided  into  groups  of  four  with  three  people  talking 
and  the  fourth  person  recording  the  speech  start  and  stop  times  (the  nodes)  of 
each  of  the  individuals  as  they  spoke.  The  communication  sequences  were  then 
graphed  and  the  audience  got  to  see  the  variation  in  communication  patterns  that 
was  occurring. 

When  ProNet  is  applied  to  communication  data  it: 

•  Can  describe  high-level  phenomena  from  low-level  data 

•  Is  very  automatic  (after  node  choice) 

•  Does  not  require  vast  data  sets 

•  Is  predictive  of  performance 

•  Could  be  used  in  real-time 


6.  Wayne  Zachary,  CHI  Systems,  Inc. 

Dr.  Wayne  Zachary  is  an  internationally  recognized  leader  in  the  human  issues 
associated  with  the  design  and  analysis  of  interactive  information  and  decision 
support  systems  in  general  and  with  cognitive  agent  design  and  development  in 
particular.  His  seminar  presentation  is  about  Cognitive  Agent  Development. 

Cognitive  Agents  and  Cognitive  Agent  Development  using  COGNET/iGEN 

Dr.  Zachary  defined  Cognitive  Agents  as  “embedded  or  self-activated  software 
entities  that  support,  train,  automate,  or  simulate  cognitive  work”.  They  are 
applied  in: 

•  Training  and  Tutoring  Systems  -  They  can  function  as  virtual  tutors,  doing 
process  assessment,  provide  coaching  and  feedback,  and  instructional 
sequencing;  they  can  act  as  synthetic  teammates  or  co-learners. 

•  Performance  Support  /Decision  Support  Systems  -  They  can  function  as 
virtual  mentors  supporting  the  decision  maker  in  managing  decision  flow, 
structuring  work  and  decision  processes,  and  keeping  the  “big  picture”; 
they  can  provide  intelligent  electronic  performance  support  by  streamlining 
scale-up,  increasing  safety,  and  reducing  liability. 

•  Engineering  and  Mission  Simulations  -  They  function  as  a  simulated 
system  user  during  design  or  acquisition,  predict  human  performance, 
and  predict  knowledge  and  training  that  system  users  will  need;  they 
function  as  synthetic  players  for  mission  analysis  or  rehearsal  simulations, 
replacing  human  role-players,  reducing  costs  and  increasing  realism. 

Following  his  comments  about  their  applicability,  he  proceeded  to  discuss  how 
cognitive  agents  are  developed.  The  “take-away”  message  from  this  discussion 
was  that  one  must  develop  agents  around  a  specific  model  of  human  information 
processing  and  knowledge  -  a  cognitive  architecture.  Specifically,  use 
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architecture  as  a  reference  point  to  collect  human  expertise,  formalize  it,  and 
execute  it.  Then,  integrate  the  ‘bottled’  expertise  with  functionality  for  agency 
(e.g.,  sensation,  action)  to  create  a  cognitive  agent. 

Dr.  Zachary  has  developed  an  agent-oriented  cognitive  modeling  framework 
(called  COGNET)  that  makes  the  connection  between  models,  theories,  and 
tools.  COGNET  models  expert  or  near-expert  levels  of  cognitive  skill  in 
sustained  real-time/multi-tasking  work.  It  is  a  model  of  how  people  think  and  act. 
Its  purpose  is  to  enable  and  simplify  the  creation  of  software  systems  (cognitive 
agents)  that  behave  like  humans  in  some  way.  It  incorporates  macro-level 
cognitive  theories  where  necessary.  iGEN  is  a  COTS  toolkit  that  includes  a 
computational  implementation  of  COGNET  and  tools  for  building  and  deploying 
cognitive  agents.  COGNET  addresses  the  following  principles  of  operation: 

•  Attention  Focus 

•  Pattern-Based  Attention  Demand 

•  Attention  Capture 

•  Task  Interruption 

•  Cognitive  Process  Modification 

•  Perceptual  Process  Modification 

•  Multiple  Task  Instances 

iGEN  ,  the  implementation  tool  for  the  COGNET  model,  provides  an  integrated 
environment  for  developing  expertise  models,  competence  models,  and 
performance  models  using  a  graphical  Integrated  Development  Environment 
(IDE).  Once  built  the  models  can  be  separated  and  executed  without  the  IDE  as 
standalone  or  embedded  cognitive  agents. 

Dr.  Zachary  provided  an  example  of  cognitive  agents  applied  to  the  domain  of  air 
traffic  control.  We  don’t  see  the  agent;  we  see  the  effects  of  it  working. 

To  get  more  details  about  COGNET  and  iGEN,  visit  this  web  site: 
www.cognitiveagent.com 
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Appendix  G.  AFOSR  Workshop  Summary 

George  Mason  University 
Fairfax,  VA 
16- 17  June  2003 

Decision  Sciences 

The  integrated  list  of  research  areas  identified  by  the  work  groups  is  presented  below: 

•  Behavior  Moderators 

•  The  Role  of  Human  Emotion  in  Mixed  Initiative  Decision  Making 

•  Adaptation  to  Human  Differences 

•  Complex,  Dynamic  Decision  Making 

•  Human  Decision  Processes  in  Military  Command  and  Control 

•  Phase  Transitions  Instrumentation 

•  Characterization  of  Decision  Landscape 

•  Measures  of  Difficulty  of  Problems 

•  Communicating  commander  (strategic)  intent 

•  Modeling  user  Intent 

•  Problem  instances  and  scenarios 

•  Research  Question  Identification 

•  Optimal  Dynamic  Task  Allocation  between  Humans  and  Computers 

•  Locus  of  Control 

•  Blending  of  H  v  M  Adaptation 

•  Hybrid  Reasoning 

•  Decision  Management 

•  Enhance  cognitive  processing 

•  Expertise 

•  Knowledge  Sharing 

•  KB  +  Reasoning 

•  Traceability  and  content  representation  for  explanation  and  analysis 

•  Visualizations  and  Uncertainty 

•  Modeling  the  vulnerabilities 

•  Dynamic  filtering  of  massive  data  sources 

•  Modeling  Team  Decision  Making 

•  Structural  Adaptation  of  models 

•  Resource  Quality  T radeoffs 

The  invited  workshop  attendees  were: 

•  Kevin  Bennett,  Wright  State  University 

•  Jerome  R.  Busemeyer,  Indiana  University 

•  Jon  Cagan,  Carnegie  Mellon  University 

•  David  Goldberg,  University  of  Illinois  at  Urbana-Champagne 

•  Carla  Gomes,  Cornell  University 

•  Wayne  D.  Gray,  Rensselaer  Polytechnic  Institute 

•  Krishna  Pattipate,  University  of  Connecticut 

•  Emilie  Roth,  Roth  Cognitive  Engineering 

•  Edward  Wegman,  George  Mason 

•  J.  Frank  Yates,  University  of  Michigan 
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Appendix  H.  Summer  Study  Presentation  Charts 


Select  Presentation  Charts: 

Supporting  Strategic  Decision  Processes  in  Adversarial  Task 

Environments 

A  Multidisciplinary  Investigation  into  Assessing  the  Functionality  of 

Information  Technology 

July  2003 

Authors:  Thomas  Triscari,  Jr.,  David  McKeen,  Gi  Lee,  William  Wales,  and  John 
White. 


Presentation  Outline 


Presentation  Will  Cover: 


[  Changing  Environment  j  [  Evolving  Strategy]  [  Technology  Development]  [  Naturalistic  Laboratories  | 


The  Environment,  Strategy,  Technology,  and  Experimentation 

The  environment  and  our  enemies  are  changing 
Strategies  must  change  to  stay  applicable 
Decision  technologies/systems  must  be  evaluated 
A  naturalistic  laboratory  may  lend  such  validation 


Presentation  Goal: 

To  highlight  the  need  to  revisit  our  decision  technology 
development  strategies  due  to  the  increased  tempo  and  uncertainty 
of  future  hypercompetitive  environments  characterized  by 
a  significantly  lower  threshold  for  failure  in  a  C2  decision  system 
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Presentation  Introduction 


Graduate  Researchers 

□  David  McKeen  (Adversarial  Task  Environments) 

■  M.S.  Information  Technology 

■  B.S.  Management  Science 

■  Prior  Experience:  CIA  Langley 

□  Gl  Lee  (Supporting  Strategic  Decision  Processes) 

■  M.B.A.  New  Product  Development 

■  B.S.  Biomedical  Engineering 

■  Prior  Experience:  Strategic  Entrepreneurial  Startups 

□  William  Wales  (Assessing  the  Functionality  of  IT) 

■  M.S.  Information  Technology 

■  B.S.  Infonnation  Technology 

■  Prior  Experience:  GE  Power  Systems  IT  Development 

□  John  White  (Multidisciplinary  Investigations) 

■  M.S.  Organizational  Psychology 

■  B.S.  General  Psychology 

■  Prior  Experience:  Naval  Air  Warfare  Center  Training  Systems  Division 


Defining  Terminology 


Presentation  Introduction 


□  Hypercompetition 

■  Multi-criteria  decisions  that  must  be  made  with  incomplete  data, 
uncertain  information,  ambiguous  goals;  high  stakes,  under  time 
pressure,  and  against  intelligent  adversary(ies). 

□  Decision  Technology 

■  Subset  of  information  technology  that  focuses  on  aiding  the  decision 
maker  in  complex  situations  through  decision  support  tools  and  aids. 

□  Decision  System 

■  System  consists  of  both  human(s)  and  machine(s)  that  work  together 
to  enhance  the  performance  capability  of  the  human  to  effectively 
develop  a  coherent  strategy  in  a  hypercompetitive  environment. 

□  Naturalistic  Laboratory 

■  A  simulated  real-world  environment  that  employs  a  multidisciplinary 
team  of  researchers  to  evaluate  decision  technologies  within  the 
context  of  a  decision  system  through  observation  &  experimentation. 
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Adversarial  Task  Environments 


□  Landscape  of  Conflict  has  Changed 

■  Threats,  Tasks,  Environments,  Information/Economic/Political  Warfare 

□  Our  Enemies  are  Fundamentally  Different 

■  Values/Beliefs,  Strategies,  Locations,  Will,  Dependencies,  Methods 

□  Changing  Enemies  &  Landscape  Yields  a  New  Environment 

■  New  Models,  Doctrine,  Organizational  Structure,  decision  technology 


Changing  Environment 


|  Evolving  Strategy")  i - |  Technology  Development^  c 


Naturalistic  Laboratories 


A  Changing  Landscape 


Our  Military  must  continue 
to  be  adaptable  &  capable  of 
recognizing  when  the  landscape 
of  conflict  has  changed 


Threats 

Tasks 

Environment 

Models 

Media 

Battle  Space 

Past 

(Cold  War) 

Few, 

well  known 
(Nation  States) 

Focused 

Superpowers 

Stable 
(U.S.S.R) 
Systems  Untested 

Many 

(Behavioral 

and 

Technical) 

Controlled 

Defined  Theater 
Defined  Enemy 
Motivations/  Values 
Structure/  Doctrine 

Present 

Many,  Many 
Unknown 
(Nation  States  & 
Organizations) 

Diverse 

Coalition 

Volatile 
(Access  to 
Knowledge) 
Systems  Tested 

Many 

(Technical) 

Few 

(Behavioral) 

Pervasive 

UnDefined  Theater 
UnDefined  Enemies 
Motivations/  Values 
Structure/  Doctrine 
Coalition  Efforts 

Future 

Many,  many 
unknown 

Greater 
expansion 
of  tasks 

Hyper- 

Competitive 

Severe  Stress 

On  Systems 

Few 

(Behavioral 
and  Technical) 

Invasive 

UnDefined  Theater 

UnDefined  Enemies 

Accelerated  Tempo 

Global  Reach 
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The  New  Face  of  Warfare 


□  Threats  are  Additive 

■  Conventional  +  New 

■  e.g.  Symmetric  &  Asymmetric 

□  Information  Based  Warfare 

■  Info  in  War  &  Info  Based  Warfare 

■  Economic,  Political,  Social  Warfare 

□  Organizational  Targeting  Philosophy 

■  A1  Qaeda  vs.  Iraq 

□  Precise  Tactical  Operations 

■  1  Weapon  =  1  Target  +  No  Collateral  Damage 

□  Effects  Based  Operations 

■  Enemy  Viewed  as  a  System 

■  Obtuse  Centers  of  Gravity 


Capabilities  must  evolve  to  match  the  constantly  changing  Threats 


Changing  Environment 


|  Evolving  Strategyj  |  Technology  Development  j  |  Naturalistic  Laboratories  j 


Area  of  Focus  in  C2  Support 


The  Managerial  Decision  Process,  Harrison,  1987 


Category  I  Decisions  (Machine) 

Category  II  Decisions  (Human) 

Decision 

Classifications 

Programmable;  routine; 
generic;  computational; 

Nonprogrammable;  creative;  adaptive 
judgmental;  inspirational;  innovative 

Decision 

Structure 

Procedural;  structured;  recurring; 
Certain  cause/effect  relationships 
Defined  information  channels 

Definite  decision  criteria 

Deductive  Reasoning 

Novel,  unstructured,  complex, 

Uncertain  cause/effect  relationships 
Incomplete  information; 

Unknown  decision  criteria 

Perceptual  Reasoning 

Employed 

Strategy 

Reliance  upon  rules  and  principles; 
habitual  reactions;  prefabricated 
response;  uniform  processing; 
computational  techniques; 
accepted  method  for  handling 

Reliance  on  judgment,  intuition,  and 
creativity;  individual  processing; 
heuristic  problem  solving 
techniques;  rules  of  thumb;  general 
problem  solving  processes 

Changing  Environment 


Evolving  Strategy 


| Technology  Development]  i - [  Naturalistic  Laboratories  ] 
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Strategy  Affects  Decision  Making 


□ 

□ 


□ 


A  Measure  of  Effectiveness 
Yield  Situational  Awareness 

■  Organization  Analysis 

■  Adversary  Analysis 

■  Environment  Analysis 

■  Rules  of  Engagement 


Decision  Task  Structure 


Degree  of  Analyzability 


Strategy  Reduces  the  Decision  Space 

■  Scope  of  Search  Space  for  Relevant  CO  As 

■  Creative  and  Critical  Thinking  Within  Strategy 


Making  Sound  and  Timely  Decisions  is  a  Key  Objective  of  the  C2  Process. 

Source:  From  Naval  Doctrine  Publication  6  -  Naval  Command  and  Control,  1995 


Changing  Environment 


Evolving  Strategy 


■  |  Technology  Development  j  [  Naturalistic  Laboratories  ] 


Assessing  the  Functionality  of  IT 

□  The  way  we  support  our  strategies  by  developing  technology  in 
this  new  environment  will  determine  our  success 

□  A  true  holistic  approach  to  system  development  takes  into 
account  aspects  of  the  task  environment,  the  organization,  and  the 
individuals  who  comprise  the  systems  being  developed 

□  Human  beings  possess  numerous  abilities  that  decision 
technologies  must  be  designed  to  enhance  and  develop 

□  Decision  Technologies  must  be  designed  to  take  into  account  and 
mitigate  human  bias,  heuristics,  and  cognitive  limitations 

□  Many  decision  technologies  focus  on  supporting  individual  tasks 
however  decision  processes  are  typically  group  based 

□  Decision  systems  must  be  evaluated  before  they  are  fielded  to 
avoid  unintended  consequences  in  mission  critical  situations 


Changing  Environment 


Evolving  Strategy 


Technology  Development 


Naturalistic  Laboratories 
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[changing 


Environment 


Evolving  Strategy  I 


Technology  Development 


Naturalistic  Laboratories 


Decision  Technology  Design 

□  Traditional  Systems  Approach 

■  Machine  Centered  Focus 

□  System  Features,  Execution 

■  User  Supported 

□  GUI,  Human  Computer  Interaction 

□  Meta- Systems  Approach 

■  Decision  System  Defined  as  Human  &  Machine 

■  Expanded  Context  in  Which  Systems  are  Evaluated 

■  Human-Machine  Context  Registration  (Enabling  Mixed  initiative) 

■  Human  Abilities  Viewed  as  Focal  to  a  User  Centric  Focus 

The  Human  is  an  integral  part  of  the  system  itself,  not  an  external  actor 
The  System  is  part  of  a  greater  organizational  framework  and  task  environment 


An  Uncertain  Environment 

"One  cannot  have  a  prepared 
list  of  rules  for  all  possible 
situations,  for  the  same  reason 
that  the  immune  system  cannot 
keep  a  list  of  all  possible 
invaders.”  -  John  Holland 


[changing  Environment]  i=i>  [  Evolving  Strategy  ] 


Technology  Development 


Naturalistic  Laboratories 


Possible  Effects  of  Technology 


Unintended  Adverse  Effects  of  IT  on  Humans 

►  ■  IT  reduces  our  will  and  need  to  learn 

■  IT  abstracts  individuals  from  the  underlying  process 

■  IT  prevents  individuals  from  finding  the  data  they  need 
•  ■  IT  discounts  individual  expertise 

■  IT  disrupts  human  pattern  recognition 

■  IT  disrupts  process  knowledge  of  how  things  work 

■  IT  can  actually  create  an  unadaptive  organization 

What  Human  Abilities  are  being  Eroded? 

■  The  ability  to  sense  of  what  are  reasonable  goals 

■  The  ability  to  detennine  trends  and  relevant  analogues 

►  ■  The  ability  to  actively  process  data  into  information  and  understanding 

■  The  ability  to  determine  what  additional  information  is  required 
-  ■  The  ability  to  notice  opportunities  /  critical  decision  points 


Gary  Klein  has  identified  human  sources  of  power  after  30+  years  of  research 
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Human  Abilities  in  Decision  Systems 

Trying  to  squeeze  more  productivity  from  a  shrinking  investment  in 
human  expertise  will  not  help  prevent  the  kinds  of  incidents  and 
accidents  that  are  currently  being  labeled  as  human  error  (FAA,  1996). 

□  Expertise,  Intuition,  Recognition 

□  Pattern  Matching,  Adaptability,  Spontaneity 

□  Learning,  Trust,  Bias,  Heuristics 

□  Perception  of  the  Environment 

■  Applicability  of  Previous  Experiences 

■  Assessing  Changes  in  the  Landscape  of  Conflict 

■  Ex.  September  1 1th  2001,  United  Airlines  Flight  93 

Understanding  how  the  human  cognitively  engages  the  decision  team  &  supporting  technology 
is  fundamental  to  developing  effective  decision  systems. 
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Defining  &  Creating  Expertise 


'One  of  the  myths  about  the  impact  of  automation  on  human  performance  is  as 
investment  in  automation  increases,  less  investment  is  needed  in  human 
expertise.  In  fact,  many  sources  have  shown  how  ^^^^^^^^^^^Bcreates 

’  -  Dr.  David  Woods 


new  knowledge  and  skill  requirements 


Amount  of  Experience  ^  Expert 
Large  Knowledgebase  ^  Expert 
Environmental  Perception  y  Expert 
High  Performance  Level  ^  Expert 

■  Area  of  a  Trapezoid 

■  Karl  Gauss  (arithmetic  series) 

□ 

n 

=  £n(n  +  1) 

*=i 


Where  do  yoir 
invest  to  create 
expertise?  Both 


Can  expertise  be 
encoded?  NO. 


Human  Centered 
.Holistic  Approach 


Expertise  is  a  function  of  knowledge,  aggregated  experiences, 
environmental  adaptability,  and  behavioral  performance 
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Team  Functions  to  Support 


□  Decision  technologies  are 
typically  designed  to 
support  individual  tasks 
however  in  many  situations 
the  underlying  decision 
process  is  team  based. 

□  Organizational 
performance  thus  relies  on 
the  effective  linking  of 
human  decision  makers  all 
supported  by  individual 
decision  aids. 


How  do  you  assess  the  functionality  of  a  decision  technology  in  this  context? 


Multidisciplinary  Investigations 

□  To  achieve  effective  evaluation  of  DSS  tools  the 
system  must  be  tested  using  experiments  in  a 
naturalistic  laboratory  environment 

□  Experiments  must  seek  to  understand  how  human 
behavior  and  cognition  effect  system  performance 

□  Experiments  must  seek  to  understand  what  impact 
the  machine  would  have  on  a  team  or  command  staff 

□  A  multidisciplinary  team  is  required  to  design,  under 
take,  and  carry  out  such  experiments 
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Multidisciplinary  Team  Importance 


□  Multi-Domain  Problems 

■  Linking  Domain  Experts 

■  Holistic  System  Development 

■  Consider  All  Env.  Factors 

■  Speed  up  Research  Cycle 

□  Reduce  Unintended 
Consequences 

■  Human(s)/Technologies 

■  Achieve  Scientific  Validity 


Cognitive 

Sciences 


Operational 
Users 


Problem 
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A  Research  Issue 


“Unintended  consequences  are  the  result  of 
focusing  80%  of  our  effort  on  intended  effects 
and  only  20%  on  unintended  while  the  reverse 
ought  to  be  the  case”  Col  Jose  Negron  GMUEBO  Workshop 


Problem: 

The  Laboratory  w/o  Naturalistic  Lab  — >  The  Field  =  Unintended  Consequences 


=  Predictive  System  Evaluation 


Designs,  Develops,  and 
Evaluate  DSS  functionality 


Scientifically  studies  Decision  Systems  in  a 
controlled  simulated  real  world  environment 


Implements  Decision  Technology  in  real  world  environment  &  gathers  user  feedback 


Solution: 

The  Laboratory  +  Naturalistic  Lab  +  The  Field 


[  Changing  Environment  ]  [  Evolving  Strategy  J  >=!>  [  Technology  Development  ]  c 
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The  Research  Cycle 


Implement 


Generate  new  ideas 
Suggest  improvements* 


USER 


Operational 

Environment 

I 


Laboratory 
/  1 


Holistic  system  testing 
Experimental  evaluations 
NATO  COBP  C2  assessment 
Risk  reduction 
Naturalistic ^^Improve  DSS  performance 
User  training 
Establish  user/tech  trust 


Send  to  real  world 


Gather  User  Feedback 
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Presentation  Take-Aways 


Presentation  Summary 


□  Adversarial  Task  Environments 

■  The  landscape  of  conflict  has  changed 

■  Our  enemies  are  fundamentally  different 

□  Supporting  Strategic  Decision  Processes 

■  Our  strategies  must  evolve  to  stay  applicable  in  a  rapidly  changing  environment 

■  Decision  systems  must  develop  strategies  to  fight  an  uncertain  environment 

□  Assessing  the  Functionality  of  Decision  Technology 

■  DSS  must  be  human  centric  and  view  the  system  as  both  human(s)  and  machine 

■  DSS  must  be  evaluated  to  avoid  unintended  consequences  in  critical  situations 

□  Multidisciplinary  Investigations 

■  Decision  systems  should  be  tested  using  scientific  experimentation 

■  A  multidisciplinary  team  is  required  to  design  and  carry  out  such  experiments 

If  the  tempo  and  uncertainty  of  a  hypercompetitive  environment  is 
yielding  a  future  where  the  room  for  error  &  threshold  for  failure 
in  a  command  and  control  decision  system  is  significantly  less. 

Then  Our  decision  technology  development  strategies  must  be  revisited. 
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Future  Recommendations 


An  Actionable  Briefing 

□  Future  Picture  (C2I  Advanced  Futurecasting  Environment  -  CAFE  Vision) 

■  Recognition  on  a  national  level  that  AFRL/IF  is  the  place  where 

□  “The  C2I  realities  of  tomorrow  are  being  created  today” 

■  A  Seamless  &  Synchronized  Organization 

■  An  Increased  Pace  of  Innovation  (Exploration,  Development,  Evaluation,  &  Transition) 

■  A  Practical  Experimental  Evaluation  Focus 

■  Enhanced  Capabilities  to  generate,  mature,  and  rapidly  deliver  decision  technologies 

□  Recommendations  /  Stepping  Stones 

■  How  do  we  invest  in  the  future? 

■  First  Institutionalize  Best  Practices  (e.g.,  TADMUS) 

■  Approach  C2  decision  technologies  as  decision  systems 

□  Employ  multidisciplinary  teams  to  evaluate  decision  performance 

□  Attract  field  commanders  and  operators  to  partake  in  training  exercises 

□  Leverage  CTCC  as  decision  system  research  and  evaluation  center 

□  Importance  of  Academia-Lab  Connection 

■  Researchers  Must  Use  Naturalistic  Subjects,  Systems,  &  Settings 

■  Buy-in  vs.  Engagement  (Go,  Listen,  Go  Home  vs.  Active  Engaging) 

■  Offsite  Evaluation  Labs  Will  Not  Significantly  Alter  Design  Practices 

“The  future  is  not  some  place  we  are  going  but  one  we  are  creating.  The  paths  to  it  are  not  found  but 
made,  and  the  activity  of  making  them  changes  both  the  maker  and  destination.”  -John  Schaar 


Key  Research  Issues 


Future  Recommendations 


□  How  do  we  develop  and  evaluate  decision  technologies  within 
the  expanded  context  of  decision  systems? 

□  How  can  a  multidisciplinary  team  be  employed  to  construct 
decision  systems  that  will  be  effective  in  a  hypercompetitive 
environment? 

□  How  can  the  scientific  research  methods  be  used  to  design, 
develop,  and  evaluate  systems  from  a  user  centered,  human 
abilities  driven  approach? 

□  What  methods  should  be  used  in  the  design/assessment  of 
decision  systems  to  reduce  adverse  consequences? 

□  What  organizational  structure,  values,  and  initiative  will  best 
facilitate  innovative  decision  technology  design? 
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